Re: [ceph-users] Scuttlemonkey signing off...

[John Wilkins]

Sorry to see you go Patrick. You've been at this as long as I have. Best of
luck to you!

On Tue, May 23, 2017 at 6:01 AM, Wido den Hollander <w...@42on.com> wrote:

> Hey Patrick,
>
> Thanks for all your work in the last 5 years! Sad to see you leave, but
> again, your effort is very much appreciated!
>
> Wido
>
> > Op 22 mei 2017 om 16:36 schreef Patrick McGarry <pmcga...@redhat.com>:
> >
> >
> > Hey cephers,
> >
> > I'm writing to you today to share that my time in the Ceph community
> > is coming to an end this year. The last five years (!!) of working
> > with the Ceph community have yielded some of the most rewarding
> > adventures of my professional career, but a new opportunity has come
> > along that I just couldn't pass up.
> >
> > I will continue to work through the end of July in order to transition
> > my responsibilities to a replacement.  In the spirit of Ceph openness,
> > I am currently assisting Stormy Peters (Red Hat's senior community
> > manager - sto...@redhat.com) in seeking candidates, so if you know
> > anyone who might be interested in managing the Ceph community, please
> > let me know.
> >
> > While this is definitely bittersweet for me, the Ceph community has
> > done a good job of self-managing, self-healing, and replicating just
> > like the underlying technology, so I know you are all in good hands
> > (each others!).  If you would like to keep in touch, or have questions
> > beyond the time I am able to answer my @redhat.com email address, feel
> > free to reach out to me at pmcga...@gmail.com and I'll be happy to
> > catch up.
> >
> > If you have any questions or concerns in the meantime feel free to
> > reach out to me directly, but I'll do my best to ensure there is
> > minimal distruption during this transition. Thank you to all of you in
> > the Ceph community who have made this journey so rewarding. I look
> > forward to seeing even more amazing things in Ceph's future!
> >
> >
> > --
> >
> > Best Regards,
> >
> > Patrick McGarry
> > Director Ceph Community || Red Hat
> > http://ceph.com  ||  http://community.redhat.com
> > @scuttlemonkey || @ceph
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
John Wilkins
Red Hat
jowil...@redhat.com
(415) 425-9599
http://redhat.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Minimum failure domain

[John Wilkins]

The classic case is when you are just trying Ceph out on a laptop (e.g.,
using file directories for OSDs, setting the replica size to 2, and
setting osd_crush_chooseleaf_type to 0).

The statement is a guideline. You could, in fact, create a CRUSH hierachy
consisting of OSD/journal groups within a host too. However, capturing the
host as a failure domain is preferred if you need to power down the host to
change a drive (assuming it's not hot-swappable).

There are cases with high density systems where you have multiple nodes in
the same chassis. So you might opt for a higher minimum failure domain in a
case like that.
There are also cases in larger clusters where you might have, for example,
three racks of servers with three top-of-rack switches--one for each rack.
If you want to isolate out the top of rack switch as a failure domain, you
will want to add the nodes/chassis to a rack within your CRUSH hierarchy,
and then select the rack level as your minimum failure domain. In those
scenarios, Ceph primary OSDs will replicate your copies to OSDs on
secondary nodes across chassis or racks respectively.

On Thu, Oct 15, 2015 at 1:55 PM, J David <j.david.li...@gmail.com> wrote:

> In the Ceph docs, at:
>
> http://docs.ceph.com/docs/master/rados/deployment/ceph-deploy-osd/
>
> It says (under "Prepare OSDs"):
>
> "Note: When running multiple Ceph OSD daemons on a single node, and
> sharing a partioned journal with each OSD daemon, you should consider
> the entire node the minimum failure domain for CRUSH purposes, because
> if the SSD drive fails, all of the Ceph OSD daemons that journal to it
> will fail too."
>
> This, of course, makes perfect sense.  But, it got me wondering...
> under what circumstances would one *not* consider a single node to be
> the minimum failure domain for CRUSH purposes?
>
> Thanks!
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
John Wilkins
Red Hat
jowil...@redhat.com
(415) 425-9599
http://redhat.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] RHEL 7 Installs

[John Wilkins]

Ken,

I had a number of issues installing Ceph on RHEL 7, which I think are
mostly due to dependencies. I followed the quick start guide, which
gets the latest major release--e.g., Firefly, Giant.

ceph.conf is here: http://goo.gl/LNjFp3
ceph.log common errors included: http://goo.gl/yL8UsM

To resolve these, I had to download and install libunwind and python-jinja2.

It also seems that the Giant repo had 0.86 and 0.87 packages for
python-ceph, and ceph-deploy didn't like that.

ceph.log error: http://goo.gl/oeKGUv

To resolve this, I had to download and install python-ceph v0.87.
Then, run the ceph-deploy install command again.


-- 
John Wilkins
Red Hat
jowil...@redhat.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Documentation of ceph pg num query

[John Wilkins]

Have you looked at

http://ceph.com/docs/master/rados/operations/monitoring-osd-pg/
http://ceph.com/docs/master/rados/operations/pg-states/
http://ceph.com/docs/master/rados/operations/pg-concepts/

On Fri, Jan 9, 2015 at 1:24 AM, Christian Eichelmann
christian.eichelm...@1und1.de wrote:
 Hi all,

 as mentioned last year, our ceph cluster is still broken and unusable.
 We are still investigating what has happened and I am taking more deep
 looks into the output of ceph pg pgnum query.

 The problem is that I can find some informations about what some of the
 sections mean, but mostly I can only guess. Is there any kind of
 documentation where I can find some explanations of whats state there?
 Because without that the output is barely usefull.

 Regards,
 Christian
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
John Wilkins
Red Hat
jowil...@redhat.com
(415) 425-9599
http://redhat.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Dependency issues in fresh ceph/CentOS 7 install

[John Wilkins]

Massimiliano,

We have a documentation update coming shortly. RHEL 7 doesn't have
yum-priorities, but you can use rpmfind to get it.

Regards,


John

On Tue, Nov 25, 2014 at 3:02 AM, Massimiliano Cuttini m...@phoenixweb.it 
wrote:
 Hi travis,

 can I have a develop account or tester account in order to submit issue by
 myself?

 Thanks,
 Massimiliano Cuttini


 Il 18/11/2014 23:03, Travis Rhoden ha scritto:

 I've captured this at http://tracker.ceph.com/issues/10133

 On Tue, Nov 18, 2014 at 4:48 PM, Travis Rhoden trho...@gmail.com wrote:

 Hi Massimiliano,

 I just recreated this bug myself.  Ceph-deploy is supposed to install EPEL
 automatically on the platforms that need it.  I just confirmed that it is
 not doing so, and will be opening up a bug in the Ceph tracker.  I'll paste
 it here when I do so you can follow it.  Thanks for the report!

  - Travis

 On Tue, Nov 18, 2014 at 4:41 PM, Massimiliano Cuttini m...@phoenixweb.it
 wrote:

 I solved by installing EPEL repo on yum.
 I think that somebody should write down in the documentation that EPEL is
 mandatory



 Il 18/11/2014 14:29, Massimiliano Cuttini ha scritto:

 Dear all,

 i try to install ceph but i get errors:

 #ceph-deploy install node1
 []
 [ceph_deploy.install][DEBUG ] Installing stable version firefly on
 cluster ceph hosts node1
 [ceph_deploy.install][DEBUG ] Detecting platform for host node1 ...
 []
 [node1][DEBUG ] --- Pacchetto libXxf86vm.x86_64 0:1.1.3-2.1.el7 settato
 per essere installato
 [node1][DEBUG ] --- Pacchetto mesa-libgbm.x86_64
 0:9.2.5-6.20131218.el7_0 settato per essere installato
 [node1][DEBUG ] --- Pacchetto mesa-libglapi.x86_64
 0:9.2.5-6.20131218.el7_0 settato per essere installato
 [node1][DEBUG ] -- Risoluzione delle dipendenze completata
 [node1][WARNIN] Errore: Pacchetto: ceph-common-0.80.7-0.el7.centos.x86_64
 (Ceph)
 [node1][WARNIN] Richiede: libtcmalloc.so.4()(64bit)
 [node1][WARNIN] Errore: Pacchetto: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)
 [node1][DEBUG ]  Si può provare ad usare --skip-broken per aggirare il
 problema
 [node1][WARNIN] Richiede: libleveldb.so.1()(64bit)
 [node1][WARNIN] Errore: Pacchetto: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)
 [node1][WARNIN] Richiede: libtcmalloc.so.4()(64bit)
 [node1][DEBUG ]  Provare ad eseguire: rpm -Va --nofiles --nodigest
 [node1][ERROR ] RuntimeError: command returned non-zero exit status: 1
 [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: yum -y
 install ceph

 I installed GIANT version not FIREFLY on admin-node.
 Is it a typo error in the config file or is it truly trying to install
 FIREFLY instead of GIANT.

 About the error, i see that it's related to wrong python default
 libraries.
 It seems that CEPH require libraries not available in the current distro:

 [node1][WARNIN] Richiede: libtcmalloc.so.4()(64bit)
 [node1][WARNIN] Richiede: libleveldb.so.1()(64bit)
 [node1][WARNIN] Richiede: libtcmalloc.so.4()(64bit)

 This seems strange.
 Can you fix this?


 Thanks,
 Massimiliano Cuttini





 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com






 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Red Hat
jowil...@redhat.com
(415) 425-9599
http://redhat.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] getting ulimit set error while installing ceph in admin node

[John Wilkins]

Subhadip,

I updated the master branch of the preflight docs here:
http://ceph.com/docs/master/start/  We did encounter some issues that
were resolved with those preflight steps.

I think it might be either requiretty or SELinux. I will keep you
posted. Let me know if it helps.

On Wed, Sep 17, 2014 at 12:13 PM, Subhadip Bagui i.ba...@gmail.com wrote:
 Hi,

 any suggestions ?

 Regards,
 Subhadip

 ---

 On Wed, Sep 17, 2014 at 9:05 AM, Subhadip Bagui i.ba...@gmail.com wrote:

 Hi

 I'm getting the below error while installing ceph in admin node. Please
 let me know how to resolve the same.


 [ceph@ceph-admin ceph-cluster]$ ceph-deploy mon create-initial ceph-admin


 [ceph_deploy.conf][DEBUG ] found configuration file at:
 /home/ceph/.cephdeploy.conf

 [ceph_deploy.cli][INFO  ] Invoked (1.5.14): /usr/bin/ceph-deploy mon
 create-initial ceph-admin

 [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-admin

 [ceph_deploy.mon][DEBUG ] detecting platform for host ceph-admin ...

 [ceph-admin][DEBUG ] connected to host: ceph-admin

 [ceph-admin][DEBUG ] detect platform information from remote host

 [ceph-admin][DEBUG ] detect machine type

 [ceph_deploy.mon][INFO  ] distro info: CentOS 6.5 Final

 [ceph-admin][DEBUG ] determining if provided host has same hostname in
 remote

 [ceph-admin][DEBUG ] get remote short hostname

 [ceph-admin][DEBUG ] deploying mon to ceph-admin

 [ceph-admin][DEBUG ] get remote short hostname

 [ceph-admin][DEBUG ] remote hostname: ceph-admin

 [ceph-admin][DEBUG ] write cluster configuration to
 /etc/ceph/{cluster}.conf

 [ceph-admin][DEBUG ] create the mon path if it does not exist

 [ceph-admin][DEBUG ] checking for done path:
 /var/lib/ceph/mon/ceph-ceph-admin/done

 [ceph-admin][DEBUG ] done path does not exist:
 /var/lib/ceph/mon/ceph-ceph-admin/done

 [ceph-admin][INFO  ] creating keyring file:
 /var/lib/ceph/tmp/ceph-ceph-admin.mon.keyring

 [ceph-admin][DEBUG ] create the monitor keyring file

 [ceph-admin][INFO  ] Running command: sudo ceph-mon --cluster ceph --mkfs
 -i ceph-admin --keyring /var/lib/ceph/tmp/ceph-ceph-admin.mon.keyring

 [ceph-admin][DEBUG ] ceph-mon: set fsid to
 a36227e3-a39f-41cb-bba1-fea098a4fc65

 [ceph-admin][DEBUG ] ceph-mon: created monfs at
 /var/lib/ceph/mon/ceph-ceph-admin for mon.ceph-admin

 [ceph-admin][INFO  ] unlinking keyring file
 /var/lib/ceph/tmp/ceph-ceph-admin.mon.keyring

 [ceph-admin][DEBUG ] create a done file to avoid re-doing the mon
 deployment

 [ceph-admin][DEBUG ] create the init path if it does not exist

 [ceph-admin][DEBUG ] locating the `service` executable...

 [ceph-admin][INFO  ] Running command: sudo /sbin/service ceph -c
 /etc/ceph/ceph.conf start mon.ceph-admin

 [ceph-admin][DEBUG ] === mon.ceph-admin ===

 [ceph-admin][DEBUG ] Starting Ceph mon.ceph-admin on ceph-admin...

 [ceph-admin][DEBUG ] failed: 'ulimit -n 32768;  /usr/bin/ceph-mon -i
 ceph-admin --pid-file /var/run/ceph/mon.ceph-admin.pid -c
 /etc/ceph/ceph.conf --cluster ceph '

 [ceph-admin][DEBUG ] Starting ceph-create-keys on ceph-admin...

 [ceph-admin][WARNIN] No data was received after 7 seconds,
 disconnecting...

 [ceph-admin][INFO  ] Running command: sudo ceph --cluster=ceph
 --admin-daemon /var/run/ceph/ceph-mon.ceph-admin.asok mon_status

 [ceph-admin][ERROR ] admin_socket: exception getting command descriptions:
 [Errno 2] No such file or directory

 [ceph-admin][WARNIN] monitor: mon.ceph-admin, might not be running yet

 [ceph-admin][INFO  ] Running command: sudo ceph --cluster=ceph
 --admin-daemon /var/run/ceph/ceph-mon.ceph-admin.asok mon_status

 [ceph-admin][ERROR ] admin_socket: exception getting command descriptions:
 [Errno 2] No such file or directory

 [ceph-admin][WARNIN] ceph-admin is not defined in `mon initial members`

 [ceph-admin][WARNIN] monitor ceph-admin does not exist in monmap

 [ceph-admin][WARNIN] neither `public_addr` nor `public_network` keys are
 defined for monitors

 [ceph-admin][WARNIN] monitors may not be able to form quorum

 [ceph_deploy.mon][INFO  ] processing monitor mon.ceph-monitor



 Regards,
 Subhadip

 ---



 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Inktank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw-admin pools list error

[John Wilkins]

Does radosgw-admin have authentication keys available and with
appropriate permissions?

http://ceph.com/docs/master/radosgw/config/#create-a-user-and-keyring

On Fri, Sep 12, 2014 at 3:13 AM, Santhosh Fernandes
santhosh.fernan...@gmail.com wrote:
 Hi,

 Anyone help me why my radosgw-admin pool list  give me this error

 #radosgw-admin pools list
 couldn't init storage provider

 But the rados lspools list all the pools,

 Regards,
 Santhosh


 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Inktank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-deploy

[John Wilkins]

No noise. I ran into the /var/local/osd0/journal issue myself. I will
add notes shortly.

On Fri, Apr 4, 2014 at 6:18 AM, Brian Candler b.cand...@pobox.com wrote:
 On 04/04/2014 14:11, Alfredo Deza wrote:

 Have you set passwordless sudo on the remote host?#

 No. Ah... I missed this bit:

 echo ceph ALL = (root) NOPASSWD:ALL | sudo tee /etc/sudoers.d/ceph
 sudo chmod 0440 /etc/sudoers.d/ceph

 The reason being that I misread the preceeding instruction:

  For other Ceph Nodes (and for initial monitors prior to ceph-deploy
 v1.1.3)...

 As I wasn't using an old version of ceph-deploy I skipped these steps. But I
 should have read it as applying to *all* ceph nodes.

 Gulp - sorry for the noise!

 Regards,

 Brian.


 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Inktank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] librados: client.admin authentication error

[John Wilkins]

Can you provide some background?

I've just reworked the cephx authentication sections. They are still in a
wip branch, and as you ask the question, it occurs to me that we do not
have a troubleshooting section for authentication issues.

It could be any number of things:

1. you don't have the client.admin key on the client where you are
executing ceph --status
2. you have a key mismatch
3. the key permissions aren't set for your user (e.g., try sudo).

The updated sections are:

http://ceph.com/docs/wip-doc-authentication/rados/configuration/auth-config-ref/
http://ceph.com/docs/wip-doc-authentication/rados/operations/user-management/

I've put the how it works theory into the architecture doc:

http://ceph.com/docs/wip-doc-authentication/architecture/#high-availability-authentication

It does strike me that we could use a bit of troubleshooting for
authentication issues.








On Wed, Aug 6, 2014 at 7:56 AM, O'Reilly, Dan daniel.orei...@dish.com
wrote:

 Anybody know why this error occurs, and a solution?



 [ceph@tm1cldcphal01 ~]$ ceph --version

 ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)

 [ceph@tm1cldcphal01 ~]$ ceph --status

 2014-08-06 08:55:13.168770 7f5527929700  0 librados: client.admin
 authentication error (95) Operation not supported

 Error connecting to cluster: Error



 Dan O'Reilly

 UNIX Systems Administration

 [image: cid:638154011@09122011-048B]

 9601 S. Meridian Blvd.

 Englewood, CO 80112

 720-514-6293





 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Inktank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph --status Missing keyring

[John Wilkins]

Dan,

Do you have /etc/ceph/ceph.client.admin.keyring, or is that in a local
directory?

Ceph will be looking for it in the /etc/ceph directory by default.

See if adding read permissions works, e.g., sudo chmod +r. You can also try
sudo when executing ceph.




On Wed, Aug 6, 2014 at 6:55 AM, O'Reilly, Dan daniel.orei...@dish.com
wrote:

 Any idea what may be the issue here?



 [ceph@tm1cldcphal01 ~]$ ceph --status

 2014-08-06 07:53:21.767255 7fe31fd1e700 -1 monclient(hunting): ERROR:
 missing keyring, cannot use cephx for authentication

 2014-08-06 07:53:21.767263 7fe31fd1e700  0 librados: client.admin
 initialization error (2) No such file or directory

 Error connecting to cluster: ObjectNotFound

 [ceph@tm1cldcphal01 ~]$ ll

 total 372

 -rw--- 1 ceph ceph 71 Aug  5 21:07 ceph.bootstrap-mds.keyring

 -rw--- 1 ceph ceph 71 Aug  5 21:07 ceph.bootstrap-osd.keyring

 -rw--- 1 ceph ceph 63 Aug  5 21:07 ceph.client.admin.keyring

 -rw--- 1 ceph ceph289 Aug  5 21:01 ceph.conf

 -rw--- 1 ceph ceph 355468 Aug  6 07:53 ceph.log

 -rw--- 1 ceph ceph 73 Aug  5 21:01 ceph.mon.keyring

 [ceph@tm1cldcphal01 ~]$ cat ceph.conf

 [global]

 auth_service_required = cephx

 filestore_xattr_use_omap = true

 auth_client_required = cephx

 auth_cluster_required = cephx

 mon_host = 10.18.201.110,10.18.201.76,10.18.201.77

 mon_initial_members = tm1cldmonl01, tm1cldmonl02, tm1cldmonl03

 fsid = 474a8905-7537-42a6-8edc-1ab9fd2ca5e4



 [ceph@tm1cldcphal01 ~]$



 Dan O'Reilly

 UNIX Systems Administration

 [image: cid:638154011@09122011-048B]

 9601 S. Meridian Blvd.

 Englewood, CO 80112

 720-514-6293





 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Inktank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Problems during first install

[John Wilkins]

?

  ceph-deploy is very verbose because some of these things are hard to
 debug. Mind sharing that output?


 Met vriendelijke groet/With kind regards,

 Tijn Buijs

 [image: Cloud.nl logo]

 t...@cloud.nl | T. 0800-CLOUDNL / +31 (0)162 820 000
 %2B31%20%280%29162%20820%2 | F. +31 (0)162 820 001
 %2B31%20%280%29162%20820%20001
 Cloud.nl B.V. | Minervum 7092D | 4817 ZK Breda | www.cloud.nl

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com





 ___
 ceph-users mailing 
 listceph-us...@lists.ceph.comhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] qemu/librbd versus qemu/kernel module rbd

[John Wilkins]

Chad,

I'm sure others can speak to performance better than I can. However, using
a kernel RBD was only vulnerable to deadlocks when trying to mount a block
device on the same host that had Ceph server daemons like monitors or osds
running. It was a kernel issue not a Ceph issue. Otherwise, the reason to
use kernel is that you intend to mount a block device on your local host
and use it. There's nothing particularly special about that, as you are
just mounting and using a block device. The cool thing about Ceph block
devices is that they are thin-provisioned and striped across a cluster. So
you could do something like mount a 100TB drive and get good performance
even though there is no such thing in the physical world as a 100TB hard
drive at this point.

Using librbd, as you've pointed out, doesn't run afoul of potential Linux
kernel deadlocks; however, you normally wouldn't encounter this type of
situation in a production cluster anyway as you'd likely never use the same
host for client and server components. The benefit of using librbd, among
other things, is that you can use it with virtual machines. That's actually
a big part of how we provide block devices to cloud computing platforms
like OpenStack.

Virtualization enables lots of scenarios. You can run multiple virtual
machines on a host, and mount block devices within those virtual machines.
A compelling scenario for cloud computing, however, is to use RBD-based
images to spin up virtual machines. In other words, you create a golden
image that you can snapshot and then use copy-on-write cloning to bring up
VMs using an RBD-based image snapshot quickly.

OS image sizes are often sizable. So downloading them each time would be
time-consuming and slow. If you can do that once and snapshot the image;
then, clone the snapshot, that's dramatically faster.

See: http://ceph.com/docs/master/rbd/rbd-snapshot/ for details on
snapshotting.

See: http://ceph.com/docs/master/rbd/rbd-openstack/ and notice that cloud
platforms generally feed Ceph block devices via QEMU and libvirt to the
cloud computing platform.

I hope this helps.


John





On Fri, Jun 20, 2014 at 6:58 AM, Chad Seys cws...@physics.wisc.edu wrote:

 Hi All,
   What are the pros and cons of running a virtual machine (with qemu-kvm)
 whose image is accessed via librbd or by mounting /dev/rbdX ?
   I've heard that the librbd method has the advantage of not being
 vulnerable
 to deadlocks due to memory allocation problems. ?
   Would one also benefit if using backported librbd to older kernels?  E.g.
 0.80 ceph with running on a 3.2.51 kernel should have bug fixes that the
 rbd
 module would not. ?
   Would one expect performance differences between librbd and module rbd?

 Thanks!
 Chad.
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Error in documentation

[John Wilkins]

I can address it, or if you want, you can fix it yourself:
http://ceph.com/docs/master/start/documenting-ceph/


On Thu, Jun 19, 2014 at 5:46 AM, george.ry...@stfc.ac.uk wrote:

  Hi,

 I’ve come across an error in the Ceph documentation, what’s the proper way
 for me to report it so that it gets fixed?



 (on
 http://ceph.com/docs/master/rados/operations/pools/#set-the-number-of-object-replicas
 “ceph osd pool set-quota {pool-name} [max-objects {obj-count}] [max_bytes
 {bytes}]” should read “ceph osd pool set-quota {pool-name} [max*_*objects
 {obj-count}] [max_bytes {bytes}]” and “ceph osd pool set-quota data
 max-objects 1” should read “ceph osd pool set-quota data max*_*objects
 1” – i.e. “_” not “-“ in “max-objects”)



 Thanks,



 George Ryall


 Scientific Computing | STFC Rutherford Appleton Laboratory | Harwell
 Oxford | Didcot | OX11 0QX

 (01235 44) 5021



 --
 Scanned by iCritical.


 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] understanding rados df statistics

[John Wilkins]

George,

I'll look into writing up some additional detail. We do have a description
for 'ceph df' here:
http://ceph.com/docs/master/rados/operations/monitoring/#checking-a-cluster-s-usage-stats


On Thu, Jun 19, 2014 at 8:07 AM, Gregory Farnum g...@inktank.com wrote:

 Yeah, the journal is a fixed size; it won't grow!


 On Thursday, June 19, 2014, george.ry...@stfc.ac.uk wrote:

  Having looked at a sample of OSDs it appears that it is indeed the case
 that for every GB of data we have 9 GB of Journal. Is this normal? Or are
 we not doing some Journal/cluster management that we should be?





 George



 *From:* Gregory Farnum [mailto:g...@inktank.com]
 *Sent:* 19 June 2014 13:53
 *To:* Ryall, George (STFC,RAL,SC)
 *Cc:* ceph-users@lists.ceph.com
 *Subject:* Re: [ceph-users] understanding rados df statistics



 The total used/available/capacity is calculated by running the syscall
 which df uses across all OSDs and summing the results. The total data
 is calculated by summing the sizes of the objects stored.



 It depends on how you've configured your system, but I'm guessing the
 markup is due to the (constant size) overhead of your journals. Or anything
 else which you might have stored on the disks besides Ceph?

 -Greg


 On Thursday, June 19, 2014, george.ry...@stfc.ac.uk wrote:

 Hi all,

 I’m struggling to understand some Ceph usage statistics and I was hoping
 someone might be able to explain them to me.



 If I run ‘rados df’ I get the following:

 # rados df

 pool name category KB  objects   clones
 degraded  unfound   rdrd KB   wrwr KB

 pool-1-  00
 00   000
 00

 pool-2-2339809 1299
 00   0  300   540600 3301
 2340798

 pool-3-409574914654
 00   0 396917256  3337952
 70296734

 pool-4-180283239332
 00   000
 22059790

 pool-5-  19310248582397
 00   0   668938102410614  5230404
 254457331

   total used  5402116076   137682

   total avail   854277445084

   total space   859679561160



 Pools 2 and 4 have a size of 2, whilst pools 3 and 5 have a size of 3.



 ‘ceph status’ tells me the following stats: “192 GB data, 134 kobjects,
 5151 GB used, 795 TB / 800 TB avail”



 The 192 GB of data is equal to the sum of the ‘KB’ column of the rados df
 data.  The used and available numbers are the same the totals given by
 rados df.



 What I don’t understand is how we have used 5,151 GB of data. Given the
 sizes of each pool I would expect it to be closer to 572 GB (sum of the
 size of each pool multiplied by pool ‘size’)   plus some overhead of some
 kind. This is a factor of 9 different. So my question is:  what have I
 missed?



 Cheers,



 George Ryall


 Scientific Computing | STFC Rutherford Appleton Laboratory | Harwell
 Oxford | Didcot | OX11 0QX

 (01235 44) 5021





 --
 Scanned by iCritical.





 --
 Software Engineer #42 @ http://inktank.com | http://ceph.com



 --
 Software Engineer #42 @ http://inktank.com | http://ceph.com

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephx authentication issue

[John Wilkins]

Did you run ceph-deploy in the directory where you ran ceph-deploy new and
ceph-deploy gatherkeys? That's where the monitor bootstrap key should be.


On Mon, Jun 16, 2014 at 8:49 AM, Fred Yang frederic.y...@gmail.com wrote:

 I'm adding three OSD nodes(36 osds in total) to existing 3-node cluster(35
 osds) using ceph-deploy, after disks prepared and OSDs activated, the
 cluster re-balanced and shows all pgs active+clean:

  osdmap e820: 72 osds: 71 up, 71 in
   pgmap v173328: 15920 pgs, 17 pools, 12538 MB data, 3903 objects
 30081 MB used, 39631 GB / 39660 GB avail
15920 active+clean

 However, the object write start having issue since the new OSDs added to
 cluster:

 2014-06-16 11:36:36.421868 osd.35 [WRN] slow request 30.317529 seconds
 old, received at 2014-06-16 11:36:06.104256: osd_op(client.5568.0:1502400
 default.5250.4_loadtest/512B_file [getxattrs,stat] 9.552a7900 e820) v4
 currently waiting for rw locks

 And from existing osd log, it seems it's having problem to authenticate
 the new OSDs (10.122.134.204 is the IP of one of new OSD nodes) :

 2014-06-16 11:38:25.281270 7f58562ce700  0 cephx: verify_reply couldn't
 decrypt with error: error decoding block for decryption
 2014-06-16 11:38:25.281288 7f58562ce700  0 -- 172.17.9.218:6811/2047255
  10.122.134.204:6831/17571 pipe(0x2891280 sd=90 :48493 s=1 pgs=3091
 cs=10 l=0 c=0x62d1840).failed verifying authorize reply


 The cephx auth list shows good to me:

 exported keyring for osd.45
 [osd.45]
 key = AQAoCp5TqBq/MhAANwclbs1nCgefNfxqqPnkZQ==
 caps mon = allow profile osd
 caps osd = allow *

 The key above does not match the keyring on osd.45.

 Anybody have any clue what might be the authentication issue here? I'm
 running Ceph 0.72.2.

 Thanks in advance,
 Fred

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] bootstrap-mds, bootstrap-osd and admin keyring not found

[John Wilkins]

Can you post your exact steps, and the ceph.log file?


On Fri, Jun 13, 2014 at 1:34 PM, Zhe Zhang zzxuany...@gmail.com wrote:

 Shayan Saeed shayansaeed93@... writes:

 
 
 
  Hi,
  I am following the standard deployment guide for ceph firefly. When I try
 to do the step 5 for collecting the key, it gives me warnings saying that
 keyrings not found for bootstrap-mds, bootstrap-osd and admin due to which
 the next step for deploying osds fail. Other people on this forum have had
 a
 similar problem in the past. How can this problem be solved?
 
 
 
  Regards,Shayan Saeed
 
 
 
 
 
 
 
 
  ___
  ceph-users mailing list
  ceph-users@...
  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 

 You could create keyrings with ceph-make-keys. Did you run ceph with
 compiling source code?



 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Fail to Block Devices and OpenStack

[John Wilkins]

|  {}
  | |
  |  name  |boot-from-rbd
  |  |
  | os-vol-host-attr:host  | None
  | |
  | os-vol-mig-status-attr:migstat | None
  | |
  | os-vol-mig-status-attr:name_id | None
  | |
  |  os-vol-tenant-attr:tenant_id  |   5a10a1fed82b45a7affaf57f814434bb
  ||
  |  size  |  1
  ||
  |  snapshot_id   | None
  |  |
  |  source_volid  | None
  |  |
  | status |   creating
  |   |
  |user_id |   90ed966837e44f91a582b73960dd848c
  |  |
  |  volume_type   | None
  |  |
 
 ++--+
 
  # cinder list
 
 +--++---+--+-+--+-+
  |  ID  | Status |  Name |
  |  Size | Volume Type | Bootable | Attached to |
 
 +--++---+--+-+--+-+
  | 30d1eee7-54d6-4911-af06-b35d2f8ef0c4 | error  | boot-from-rbd |  1
  |   | None|  false   | |
 
 +--++---+--+-+--+-+
 
  I've done all the setting of
  URL(http://ceph.com/docs/next/rbd/rbd-openstack/)
  There is a setup required except URL?
 
 
 
  Best Regards.
 
  Yamashita
  ___
  ceph-users mailing list
  ceph-users@lists.ceph.com
  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
  ___
  ceph-users mailing list
  ceph-users@lists.ceph.com
  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Moving Ceph cluster to different network segment

[John Wilkins]

Fred,

I'm not sure it will completely answer your question, but I would
definitely have a look at:
http://ceph.com/docs/master/rados/operations/add-or-rm-mons/#changing-a-monitor-s-ip-address

There are some important steps in there for monitors.


On Wed, Jun 11, 2014 at 12:08 PM, Fred Yang frederic.y...@gmail.com wrote:

 We need to move Ceph cluster to different network segment for
 interconnectivity between mon and osc, anybody has the procedure regarding
 how that can be done? Note that the host name reference will be changed, so
 originally the osd host referenced as cephnode1, in the new segment it will
 be cephnode1-n.

 Thanks,
 Fred

 Sent from my Samsung Galaxy S3

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph pgs stuck inactive since forever

[John Wilkins]

I'll update the docs to incorporate the term incomplete. I believe this
is due to an inability to complete backfilling. Your cluster is nearly
full. You indicated that you installed Ceph. Did you store data in the
cluster? Your usage indicates that you have used 111GB of 125GB. So you
only have about 8GB left. Did it ever get to an active + clean state?


On Wed, Jun 11, 2014 at 6:08 AM, akhil.labudubar...@ril.com wrote:

  I installed ceph and then I was ceph health it gives me the following
 output



 *HEALTH_WARN 384 pgs incomplete; 384 pgs stuck inactive;
 384 pgs stuck unclean; 2 near full osd(s)*



 This is the output of a single pg when I use ceph health detail



 *pg 2.2 is incomplete, acting [0] (reducing pool rbd min_size from 2 may
 help; search ceph.com/docs http://ceph.com/docs for 'incomplete')*



 and similar line comes up for all the pgs.



 This is the output of ceph - s



 *cluster 89cbb30c-023b-4f8b-ac14-abc78fb6b07a*

 * health HEALTH_WARN 384 pgs incomplete; 384 pgs stuck inactive; 384
 pgs stuck unclean; 2 near full osd(s)*

 * monmap e1: 1 mons at {a=100.112.12.28:6789/0
 http://100.112.12.28:6789/0}, election epoch 2, quorum 0 a*

 * osdmap e5: 2 osds: 2 up, 2 in*

 *  pgmap v64: 384 pgs, 3 pools, 0 bytes data, 0 objects*

 *111 GB used, 8346 MB / 125 GB avail*

 * 384 incomplete*


 *Confidentiality Warning*: This message and any attachments are intended
 only for the use of the intended recipient(s), are confidential and may be
 privileged. If you are not the intended recipient, you are hereby notified
 that any review, re-transmission, conversion to hard copy, copying,
 circulation or other use of this message and any attachments is strictly
 prohibited. If you are not the intended recipient, please notify the sender
 immediately by return email and delete this message and any attachments
 from your system.

 *Virus Warning:* Although the company has taken reasonable precautions to
 ensure no viruses are present in this email. The company cannot accept
 responsibility for any loss or damage arising from the use of this email or
 attachment.

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] perplexed by unmapped groups on fresh firefly install

[John Wilkins]

Miki,

osd crush chooseleaf type is set to 1 by default, which means that it looks
to peer with placement groups on another node, not the same node. You would
need to set that to 0 for a 1-node cluster.

John


On Sun, Jun 8, 2014 at 10:40 PM, Miki Habryn dic...@rcpt.to wrote:

 I set up a single-node, dual-osd cluster following the Quick Start on
 ceph.com with Firefly packages, adding osd pool default size = 2.
 All of the pgs came up in active+remapped or active+degraded status. I
 read up on tunables and set them to optimal, to no result, so I added
 a third osd instead. About 39 pgs moved to active status, but the rest
 stayed in active+remapped or active+degraded. When I raised the
 replication level to 3 with ceph osd pool set ... size 3, all the
 pgs went back to degraded or remapped. Just for kicks, I tried to set
 the replication level to 1, and I still only got 39 pgs active. Is
 there something obvious I'm doing wrong?

 m.
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Hard drives of different sizes.

[John Wilkins]

 size/min_size #size

 

  See docs http://ceph.com/docs/master/rados/operations/pools/ for

  alterations to pool attributes.

 

  -Michael

 

 

  On 05/06/2014 17:29, Vadim Kimlaychuk wrote:

 

  

 

  I have

osd pool default size = 2

  at my ceph.conf. Shouldn' it tell ceph to use 2 OSDs ? Or it is

  somewhere in CRUSH map?

 

  Vadim

  

  From: Christian Balzer [ch...@gol.com]

  Sent: Thursday, June 05, 2014 18:26

  To: Vadim Kimlaychuk

  Cc: ceph-users@lists.ceph.com

  Subject: Re: [ceph-users] Hard drives of different sizes.

 

  Hello,

 

  On Thu, 5 Jun 2014 14:11:47 + Vadim Kimlaychuk wrote:

 

  Hello,

 

   Probably this is anti-pattern, but I have to get answer

  how this will work / not work. Input:

   I have single host for tests with ceph 0.80.1 and 2 OSD:

   OSD.0 – 1000 Gb

   OSD.1 – 750 Gb

 

   Recompiled CRUSH map to set „step chooseleaf firstn 0

  type osd“

 

  You got it half right.

 

  Version .8x aka Firefly has a default replication of 3, so you would

  need

  3 OSDs at least.

 

  Christian

 

   I am expecting, that part of PG-s will have status

  „active+clean“ (with size of ~750Gb) another part of PG-s will have

  „active+degradated“ (with size of ~250Gb), because there is not

  enough place to replicate data on the second OSD.

 

   Instead I have ALL PG-s „active + degradated“

 

  Output:

health HEALTH_WARN 192 pgs degraded; 192 pgs stuck unclean

monmap e1: 1 mons at {storage=172.16.3.2:6789/0}, election

  epoch 2, quorum 0 storage osdmap e15: 2 osds: 2 up, 2 in

 pgmap v29: 192 pgs, 3 pools, 0 bytes data, 0 objects

   71496 kB used, 1619 GB / 1619 GB avail

192 active+degraded

 

   What is the logic behind this?? Can I use different

  hard drives successfully? If yes – how?

 

  Thank you for explanation,

 

  Vadim

 

 

  --

  Christian BalzerNetwork/Systems Engineer

  ch...@gol.com   Global OnLine Japan/Fusion Communications

  http://www.gol.com/

  ___

  ceph-users mailing list

  ceph-users@lists.ceph.com

  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

 

 

  ___

  ceph-users mailing list

  ceph-users@lists.ceph.com

  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

 ___

 ceph-users mailing list

 ceph-users@lists.ceph.com

 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com






 --

 Vincenzo Pii

 Researcher, InIT Cloud Computing Lab
 Zurich University of Applied Sciences (ZHAW)
 http://www.cloudcomp.ch/

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Erasure coding

[John Wilkins]

I have also added a big part of Loic's discussion of the architecture into
the Ceph architecture document here:

http://ceph.com/docs/master/architecture/#erasure-coding


On Mon, May 19, 2014 at 5:35 AM, yalla.gnan.ku...@accenture.com wrote:

 Hi Loic,

 Thanks for the reply.


 Thanks
 Kumar


 -Original Message-
 From: Loic Dachary [mailto:l...@dachary.org]
 Sent: Monday, May 19, 2014 6:04 PM
 To: Gnan Kumar, Yalla; ceph-users@lists.ceph.com
 Subject: Re: [ceph-users] Erasure coding

 Hi,

 The general idea to preserve resilience but save space compared to
 replication. It costs more in terms of CPU and network. You will find a
 short introduction here :


 https://wiki.ceph.com/Planning/Blueprints/Dumpling/Erasure_encoding_as_a_storage_backend

 https://wiki.ceph.com/Planning/Blueprints/Firefly/Erasure_coded_storage_backend_%28step_3%29

 For the next Ceph release Pyramid Codes will help reduce the bandwidth
 requirements
 https://wiki.ceph.com/Planning/Blueprints/Giant/Pyramid_Erasure_Code

 Cheers

 On 19/05/2014 13:52, yalla.gnan.ku...@accenture.com wrote:
  Hi All,
 
 
 
  What exactly is erasure coding and why is it used in ceph ? I could not
 get enough explanatory information from the documentation.
 
 
 
 
 
  Thanks
 
  Kumar
 
 
 
 --
 
  This message is for the designated recipient only and may contain
 privileged, proprietary, or otherwise confidential information. If you have
 received it in error, please notify the sender immediately and delete the
 original. Any other use of the e-mail by you is prohibited. Where allowed
 by local law, electronic communications with Accenture and its affiliates,
 including e-mail and instant messaging (including content), may be scanned
 by our systems for the purposes of information security and assessment of
 internal compliance with Accenture policy.
 
 __
 
  www.accenture.com
 
 
  ___
  ceph-users mailing list
  ceph-users@lists.ceph.com
  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 

 --
 Loïc Dachary, Artisan Logiciel Libre


 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mon create error

[John Wilkins]

Have you tried specifying the socket path in your Ceph configuration file?


On Sat, May 17, 2014 at 9:38 AM, reistlin87 reistli...@yandex.ru wrote:

 Hi all! Sorry for my english, I am russian)

 We get the same error on diffrent linux distro(CentOS 6.4  SuSe 11),
 diffrent ceph version (0.67,0.72,0.8).

 Point of error:

 We want to create new cluster with non-standart name(for example cephtst):
 [root@admin ceph]# ceph-deploy --cluster cephtst new mon
 Cluster creates Ok.

 And then we want to create monitor:
 [root@admin ceph]# ceph-deploy --cluster cephtst mon create mon

 We get a error related to name of admin socket:
 [root@admin ceph]# ceph-deploy --cluster cephtst mon create mon
 [ceph_deploy.conf][DEBUG ] found configuration file at:
 /root/.cephdeploy.conf
 [ceph_deploy.cli][INFO  ] Invoked (1.5.1): /usr/bin/ceph-deploy --cluster
 cephtst mon create mon
 [ceph_deploy.mon][DEBUG ] Deploying mon, cluster cephtst hosts mon
 [ceph_deploy.mon][DEBUG ] detecting platform for host mon ...
 [mon][DEBUG ] connected to host: mon
 [mon][DEBUG ] detect platform information from remote host
 [mon][DEBUG ] detect machine type
 [ceph_deploy.mon][INFO  ] distro info: CentOS 6.5 Final
 [mon][DEBUG ] determining if provided host has same hostname in remote
 [mon][DEBUG ] get remote short hostname
 [mon][DEBUG ] deploying mon to mon
 [mon][DEBUG ] get remote short hostname
 [mon][DEBUG ] remote hostname: mon
 [mon][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
 [mon][DEBUG ] create the mon path if it does not exist
 [mon][DEBUG ] checking for done path: /var/lib/ceph/mon/cephtst-mon/done
 [mon][DEBUG ] create a done file to avoid re-doing the mon deployment
 [mon][DEBUG ] create the init path if it does not exist
 [mon][DEBUG ] locating the `service` executable...
 [mon][INFO  ] Running command: /sbin/service ceph -c
 /etc/ceph/cephtst.conf start mon.mon
 [mon][DEBUG ] === mon.mon ===
 [mon][DEBUG ] Starting Ceph mon.mon on mon...already running
 [mon][INFO  ] Running command: ceph --cluster=cephtst --admin-daemon
 /var/run/ceph/cephtst-mon.mon.asok mon_status
 [mon][ERROR ] admin_socket: exception getting command descriptions: [Errno
 2] No such file or directory
 [mon][WARNIN] monitor: mon.mon, might not be running yet
 [mon][INFO  ] Running command: ceph --cluster=cephtst --admin-daemon
 /var/run/ceph/cephtst-mon.mon.asok mon_status
 [mon][ERROR ] admin_socket: exception getting command descriptions: [Errno
 2] No such file or directory
 [mon][WARNIN] monitor mon does not exist in monmap
 [mon][WARNIN] neither `public_addr` nor `public_network` keys are defined
 for monitors
 [mon][WARNIN] monitors may not be able to form quorum
 Unhandled exception in thread started by
 Error in sys.excepthook:
 Original exception was:

 And in this time in folder /var/run/ceph/ present file with name
 ceph-mon.mon.asok

 Why does name of admin socket not changes to right?
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mkcephfs questions

[John Wilkins]

 wrote:
  Thanks your reply, Haomai. What I don't understand is that, why the
 stuck unclean pgs keep the same numbers after 12 hours. It's the common
 behavior or not?
 
 
  Wei Cao (Buddy)
 
  -Original Message-
  From: Haomai Wang [mailto:haomaiw...@gmail.com]
  Sent: Wednesday, April 30, 2014 11:36 AM
  To: Cao, Buddy
  Cc: ceph-users@lists.ceph.com
  Subject: Re: [ceph-users] mkcephfs questions
 
  The result of ceph -s should tell you the reason. There only
  exists
  21 OSD up but we need 24 OSDs
 
  On Wed, Apr 30, 2014 at 11:21 AM, Cao, Buddy buddy@intel.com
 wrote:
  Hi,
 
 
 
  I setup ceph cluster thru mkcephfs command, after I enter “ceph
  –s”, it always returns 4950 stuck unclean pgs. I tried the same “ceph
 -s”
  after 12 hrs,  there still returns the same unclean pgs number,
 nothing changed.
  Does mkcephfs always has the problem or I did something wrong? I
  attached the result of “ceph -s”, “ceph osd tree” and ceph.conf I
  have, please kindly help.
 
 
 
 
 
  [root@ceph]# ceph -s
 
  cluster 99fd4ff8-0fb8-47b9-8179-fefbba1c2503
 
   health HEALTH_WARN 4950 pgs degraded; 4950 pgs stuck unclean;
  recovery
  21/42 objects degraded (50.000%); 3/24 in osds are down; clock skew
  detected on mon.1, mon.2
 
   monmap e1: 3 mons at
  {0=192.168.0.2:6789/0,1=192.168.0.3:6789/0,2=192.168.0.4:6789/0},
  election epoch 6, quorum 0,1,2 0,1,2
 
   mdsmap e4: 1/1/1 up {0=0=up:active}
 
   osdmap e6019: 24 osds: 21 up, 24 in
 
pgmap v16445: 4950 pgs, 6 pools, 9470 bytes data, 21 objects
 
  4900 MB used, 93118 MB / 98019 MB avail
 
  21/42 objects degraded (50.000%)
 
  4950 active+degraded
 
 
 
  [root@ceph]# ceph osd tree //part of returns
 
  # idweight  type name   up/down reweight
 
  -36 25  root vsm
 
  -31 3.2 storage_group ssd
 
  -16 3   zone zone_a_ssd
 
  -1  1   host vsm2_ssd_zone_a
 
  2   1   osd.2   up  1
 
  -6  1   host vsm3_ssd_zone_a
 
  10  1   osd.10  up  1
 
  -11 1   host vsm4_ssd_zone_a
 
  18  1   osd.18  up  1
 
  -21 0.0 zone zone_c_ssd
 
  -26 0.0 zone zone_b_ssd
 
  -33 3.2 storage_group sata
 
  -18 3   zone zone_a_sata
 
  -3  1   host vsm2_sata_zone_a
 
  1   1   osd.1   up  1
 
  -8  1   host vsm3_sata_zone_a
 
  9   1   osd.9   up  1
 
  -13 1   host vsm4_sata_zone_a
 
  17  1   osd.17  up  1
 
  -23 0.0 zone zone_c_sata
 
  -28 0.0 zone zone_b_sata
 
 
 
 
 
  Wei Cao (Buddy)
 
 
 
 
  ___
  ceph-users mailing list
  ceph-users@lists.ceph.com
  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
 
 
  --
  Best Regards,
 
  Wheat
 
 
 
  --
  Best Regards,
 
  Wheat
 
 
 
  --
  Best Regards,
 
  Wheat



 --
 Best Regards,

 Wheat

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] About ceph.conf

[John Wilkins]

Buddy,

There are significant changes between mkcephfs and ceph-deploy. The
mkcephfs script is fairly antiquated now and you should be using
ceph-deploy or some other method of deployment in our newer releases.

The mkcephfs script would read the ceph.conf file during deployment and
bootstrap monitors from information gleaned in ceph.conf. We don't deploy
Ceph that way anymore. The ceph.conf file is a great place to put
configuration settings, but it's not what we use for deployment now. One of
the reasons for that change is to ensure consistency requirements. If you
read
http://ceph.com/docs/master/rados/operations/add-or-rm-mons/#changing-a-monitor-s-ip-address
you can see why we made some of these changes. Monitors find other monitors
using the monmap, not ceph.conf.

You can view your runtime configuration. See
http://ceph.com/docs/master/rados/configuration/ceph-conf/#viewing-a-configuration-at-runtime
You can also make runtime changes.
http://ceph.com/docs/master/rados/configuration/ceph-conf/#runtime-changes

To understand what mkcephfs and ceph-deploy are doing, you can look at a
manual process for bootstrapping a cluster here:
http://ceph.com/docs/master/install/manual-deployment/

I hope this helps.








On Mon, May 5, 2014 at 9:52 PM, Cao, Buddy buddy@intel.com wrote:

  According to the change of ceph-deploy from mkcephfs, I feel ceph.conf
 is not a recommended way to manage ceph configuration. Is it true? If so,
 how do I get the configurations previous configured in ceph.conf? e.g.,
 data drive, journal drive, [osd] conf, etc.





 Wei Cao (Buddy)



 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RadosGW Quotas

[John Wilkins]

Derek,

I have some. I'll get them by the end of the week at the latest.


On Mon, Apr 7, 2014 at 4:52 PM, Yehuda Sadeh yeh...@inktank.com wrote:

 On Mon, Apr 7, 2014 at 4:34 PM, Derek Yarnell de...@umiacs.umd.edu
 wrote:
  Hi,
 
  Most likely Yehuda can speak to this.  So there is some documentation on
  the way to set quotas for a user in master (src/rgw/rgw_rest_user.cc
  line 712) but these are not in the docs it seems yet. I have started to
  incorporate them but I don't see anything as an example for how to
  document the JSON body.  Is there a convention that I should try to
 follow?

 The geo-replication related api has some requests with JSON body:

 http://wiki.ceph.com/Development/RESTful_API_for_DR_%2F%2F_Geo-Replication
 
  Second question relates to the bucket quotas.  In the example above
  these are set via the uid=$uid and quota-type parameters.  This means
  that this applies to all buckets the user owns I would expect (since I
  am not required to give a bucket name).  Is the current design that a
  bucket can't have a independent quota?
 

 Buckets can have independent quota, it just cannot be set using this
 specific api (which is used to control user info hence the api entry
 point there is /admin/user). The bucket specific quota can be set
 either through radosgw-admin, or by using the metadata api.

 Yehuda
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] first installation, ceph never goes to health ok

[John Wilkins]

Actually, it is. We took the single host getting started out, because
nobody would really deploy a distributed system like Ceph for production on
single host. The problem is that the default crush rule is set to the host
level, not the osd level.

Note, I think ceph-deploy mon create-initial will do the next two steps for
you. So those may be redundant.

What you need to do though is after you do ceph-deploy new ceph-a1, is you
need to add the following to your ceph.conf file:

osd crush chooseleaf type = 0

Then, follow the rest of the procedure.


On Fri, Jan 31, 2014 at 2:41 PM, Cristian Falcas cristi.fal...@gmail.comwrote:

 Hi list,

 I'm trying to play with ceph, but I can't get the machine to reach a
 clean state.

 How I did the installation:

 ceph-deploy new ceph-a1
 ceph-deploy install ceph-a1
 ceph-deploy mon create-initial
 ceph-deploy mon create ceph-a1
 ceph-deploy gatherkeys ceph-a1

 ceph-deploy disk zap ceph-a1:vdb ceph-a1:vdc ceph-a1:vdd ceph-a1:vde
 ceph-deploy osd prepare ceph-a1:vdb ceph-a1:vdc ceph-a1:vdd ceph-a1:vde
 ceph-deploy osd activate ceph-a1:/dev/vdb ceph-a1:/dev/vdc
 ceph-a1:/dev/vdd ceph-a1:/dev/vde

 What the status is:
 [root@ceph-a1 ~]# ceph health
 HEALTH_WARN 49 pgs degraded; 192 pgs stuck unclean

 cceph -w:
 2014-01-31 17:39:44.060937 mon.0 [INF] pgmap v25: 192 pgs: 102 active,
 41 active+remapped, 49 active+degraded; 0 bytes data, 143 MB used, 243
 GB / 243 GB avail


 Even if I add more disks or play with the crush map settings, I can't
 seem to manage to bring the PGs to a clean state.

 Is this expected with one host only?

 Best regards,
 Cristian Falcas
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Which API to use to write and read objects

[John Wilkins]

Randy,

Use librados. If you want to test out my latest doc and provide some
feedback, I'd appreciate it:
http://ceph.com/docs/wip-doc-librados-intro/rados/api/librados-intro/


On Mon, Jan 13, 2014 at 11:40 PM, Randy Breunling rbreunl...@gmail.comwrote:

 New to CEPH...so I'm on the learning-curve here.
 Have been through a lot of the documentation but am confused on one thing,
 still.

 What is the API or interface to use if we just want to write and read
 objects to CEPH object storage...and don't necessarily care about
 compatibility with Amazon s3 or Openstack SWIFT?
 Do we librados?

 Or maybe asked a different way...when would someone want or need to use
 the s3- or SWIFT-compatible API interfaces to CEPH (RADOS).

 Thanks...

 --Randy

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mon not binding to public interface

[John Wilkins]

Jeff,

First, if you've specified the public and cluster networks in [global], you
don't need to specify it anywhere else. If you do, they get overridden.
That's not the issue here. It appears from your ceph.conf file that you've
specified an address on the cluster network. Specifically, you specified mon
addr = 10.100.10.1:6789, but you indicated elsewhere that this IP address
belongs to the cluster network.


On Mon, Jan 13, 2014 at 11:29 AM, Jeff Bachtel 
jbach...@bericotechnologies.com wrote:

 I've got a cluster with 3 mons, all of which are binding solely to a
 cluster network IP, and neither to 0.0.0.0:6789 nor a public IP. I hadn't
 noticed the problem until now because it makes little difference in how I
 normally use Ceph (rbd and radosgw), but now that I'm trying to use cephfs
 it's obviously suboptimal.

 [global]
   auth cluster required = cephx
   auth service required = cephx
   auth client required = cephx
   keyring = /etc/ceph/keyring
   cluster network = 10.100.10.0/24
   public network = 10.100.0.0/21
   public addr = 10.100.0.150
   cluster addr = 10.100.10.1
fsid = de10594a-0737-4f34-a926-58dc9254f95f

 [mon]
   cluster network = 10.100.10.0/24
   public network = 10.100.0.0/21
   mon data = /var/lib/ceph/mon/mon.$id

 [mon.controller1]
   host = controller1
   mon addr = 10.100.10.1:6789
   public addr = 10.100.0.150
   cluster addr = 10.100.10.1
   cluster network = 10.100.10.0/24
   public network = 10.100.0.0/21

 And then with /usr/bin/ceph-mon -i controller1 --debug_ms 12 --pid-file
 /var/run/ceph/mon.controller1.pid -c /etc/ceph/ceph.conf I get in logs

 2014-01-13 14:19:13.578458 7f195e6d97a0  0 ceph version 0.72.2 (
 a913ded2ff138aefb8cb84d347d72164099cfd60), process ceph-mon, pid 7559
 2014-01-13 14:19:13.641639 7f195e6d97a0 10 -- :/0 rank.bind
 10.100.10.1:6789/0
 2014-01-13 14:19:13.641668 7f195e6d97a0 10 accepter.accepter.bind
 2014-01-13 14:19:13.642773 7f195e6d97a0 10 accepter.accepter.bind bound to
 10.100.10.1:6789/0
 2014-01-13 14:19:13.642800 7f195e6d97a0  1 -- 10.100.10.1:6789/0 learned
 my addr 10.100.10.1:6789/0
 2014-01-13 14:19:13.642808 7f195e6d97a0  1 accepter.accepter.bind
 my_inst.addr is 10.100.10.1:6789/0 need_addr=0

 Whith no mention of public addr (10.100.2.1) or public network (
 10.100.0.0/21) found. mds (on this host) and osd (on other hosts) bind to
 0.0.0.0 and a public IP, respectively.

 At this point public/cluster addr/network are WAY overspecified in
 ceph.conf, but the problem appeared with far less specification.

 Any ideas? Thanks,

 Jeff
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mon not binding to public interface

[John Wilkins]

Monitors use the public network, not the cluster network. Only OSDs use the
cluster network. The purpose of the cluster network is that OSDs do a lot
of heartbeat checks, data replication, recovery, and rebalancing. So the
cluster network will see more traffic than the front end public network.
See http://ceph.com/docs/master/rados/configuration/mon-osd-interaction/ By
contrast, Ceph clients connect to monitors and OSDs, so they must be on the
public network. See the diagram here:
http://ceph.com/docs/master/rados/configuration/network-config-ref/  Notice
that all daemons use the public network? This is because clients connect
using the public network. Yet, only OSDs use the cluster network.

In your configuration, you specified the following:

[mon.controller1]
  host = controller1
  mon addr = 10.100.10.1:6789
  public addr = 10.100.0.150
  cluster addr = 10.100.10.1
  cluster network = 10.100.10.0/24
  public network = 10.100.0.0/21

The IP address for the mon.controller1 is set to a cluster network IP
address--namely, 10.100.10.1:6789.  Since the monitor only connects on the
public network, and you have specifically told it to connect only on a
cluster network, that is why the monitor is running on the cluster network.
Your monitor address should be something like 10.100.0.155:6789 in that
range.

However, now that you have a monitor IP address, changing it can be a bit
troublesome too. See the following:

http://ceph.com/docs/master/rados/operations/add-or-rm-mons/#changing-a-monitor-s-ip-address





On Wed, Jan 15, 2014 at 1:13 PM, Jeff Bachtel 
jbach...@bericotechnologies.com wrote:

  If I understand correctly then, I should either not specify mon addr or
 set it to an external IP?

 Thanks for the clarification,

 Jeff


 On 01/15/2014 03:58 PM, John Wilkins wrote:

 Jeff,

  First, if you've specified the public and cluster networks in [global],
 you don't need to specify it anywhere else. If you do, they get overridden.
 That's not the issue here. It appears from your ceph.conf file that you've
 specified an address on the cluster network. Specifically, you specified mon
 addr = 10.100.10.1:6789, but you indicated elsewhere that this IP address
 belongs to the cluster network.


 On Mon, Jan 13, 2014 at 11:29 AM, Jeff Bachtel 
 jbach...@bericotechnologies.com wrote:

 I've got a cluster with 3 mons, all of which are binding solely to a
 cluster network IP, and neither to 0.0.0.0:6789 nor a public IP. I
 hadn't noticed the problem until now because it makes little difference in
 how I normally use Ceph (rbd and radosgw), but now that I'm trying to use
 cephfs it's obviously suboptimal.

 [global]
   auth cluster required = cephx
   auth service required = cephx
   auth client required = cephx
   keyring = /etc/ceph/keyring
   cluster network = 10.100.10.0/24
   public network = 10.100.0.0/21
   public addr = 10.100.0.150
   cluster addr = 10.100.10.1
fsid = de10594a-0737-4f34-a926-58dc9254f95f

 [mon]
   cluster network = 10.100.10.0/24
   public network = 10.100.0.0/21
   mon data = /var/lib/ceph/mon/mon.$id

 [mon.controller1]
   host = controller1
   mon addr = 10.100.10.1:6789
   public addr = 10.100.0.150
   cluster addr = 10.100.10.1
   cluster network = 10.100.10.0/24
   public network = 10.100.0.0/21

 And then with /usr/bin/ceph-mon -i controller1 --debug_ms 12 --pid-file
 /var/run/ceph/mon.controller1.pid -c /etc/ceph/ceph.conf I get in logs

 2014-01-13 14:19:13.578458 7f195e6d97a0  0 ceph version 0.72.2
 (a913ded2ff138aefb8cb84d347d72164099cfd60), process ceph-mon, pid 7559
 2014-01-13 14:19:13.641639 7f195e6d97a0 10 -- :/0 rank.bind
 10.100.10.1:6789/0
 2014-01-13 14:19:13.641668 7f195e6d97a0 10 accepter.accepter.bind
 2014-01-13 14:19:13.642773 7f195e6d97a0 10 accepter.accepter.bind bound
 to 10.100.10.1:6789/0
 2014-01-13 14:19:13.642800 7f195e6d97a0  1 -- 10.100.10.1:6789/0 learned
 my addr 10.100.10.1:6789/0
 2014-01-13 14:19:13.642808 7f195e6d97a0  1 accepter.accepter.bind
 my_inst.addr is 10.100.10.1:6789/0 need_addr=0

 Whith no mention of public addr (10.100.2.1) or public network (
 10.100.0.0/21) found. mds (on this host) and osd (on other hosts) bind
 to 0.0.0.0 and a public IP, respectively.

 At this point public/cluster addr/network are WAY overspecified in
 ceph.conf, but the problem appeared with far less specification.

 Any ideas? Thanks,

 Jeff
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




  --
 John Wilkins
 Senior Technical Writer
 Intank
 john.wilk...@inktank.com
 (415) 425-9599
 http://inktank.com



 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph

Re: [ceph-users] documentation comment

[John Wilkins]

Thanks Wolfgang. The fix should be in shortly.


On Thu, Jan 9, 2014 at 3:52 AM, Wolfgang Hennerbichler wo...@wogri.comwrote:

 I think I found a comment in the documentation that's not inteded to be
 there:

 http://ceph.com/docs/master/rbd/rbd-snapshot/
 For the rollback section, you could mention that rollback means
 overwriting the current version with data from a snapshot, and takes
 longer with larger images. So cloning is preferable for fast recovery.
 - this seems to be an internal comment.

 wogri

 --
 http://www.wogri.com
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] shutting down for maintenance

[John Wilkins]

Most production clusters are large enough that you don't have to bring down
the entire cluster to do maintenance on particular machines. If your
reconfiguring the entire network, that's a bit more involved. I'm not sure
what your cluster looks like, so I can't advise.  However, you mention
changing IP addresses. Changing the IP addresses for the OSDs is okay, but
you want to be careful when changing them for monitors. See
http://ceph.com/docs/master/rados/operations/add-or-rm-mons/#changing-a-monitor-s-ip-address.
 Since monitors use the public network, this shouldn't be a problem in your
case. You can change the config files if you've include OSD entries in your
config file. You may also change the settings at runtime. See
http://ceph.com/docs/master/rados/configuration/ceph-conf/#runtime-changes.


On Tue, Dec 31, 2013 at 9:35 AM, Scottix scot...@gmail.com wrote:

 The way I have done it is so the osd don't get set out.

 Check the link below


 http://ceph.com/docs/master/rados/troubleshooting/troubleshooting-osd/#stopping-w-out-rebalancing


 On Tue, Dec 31, 2013 at 12:43 AM, James Harper 
 james.har...@bendigoit.com.au wrote:

 I need to shut down ceph for maintenance to make some hardware changes.
 Is it sufficient to just stop all services on all nodes, or is there a way
 to put the whole cluster into standby or something first?

 And when things come back up, IP addresses on the cluster network will be
 different (public network will not change though). Is it sufficient to just
 change the config files and the osd's will register themselves correctly,
 or is there more involved?

 Thanks

 James
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




 --
 Follow Me: @Scottix http://www.twitter.com/scottix
 http://about.me/scottix
 scot...@gmail.com

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph primetime

[John Wilkins]

Ceph is more than just a file system. The storage cluster provides a native
API for object storage, which is the core of Ceph. Most production systems
use the Ceph Block Device (as a backend for OpenStack, CloudStack, and
other cloud platforms) or Ceph Object Storage (S3 and Swift APIs). There
are people using the filesystem for production data, and there have been
many improvements. I would suggest referring to the release notes for the
latest changes.


On Thu, Dec 26, 2013 at 5:14 AM, Keith Brown keith6...@gmail.com wrote:

 Hello,

 We are thinking of deploying a distributed filesystem for our lab. At the
 moment we are using Dell storage for a 20 node computational cluster. We
 are having many bottlenecks in storage i/o.

 We are interesting in Lustre, XtremeFS, and Ceph. It seems Ceph isn't
 ready for production according to the documentation.

 My questions are: Are people using Ceph filesystem (POSIX) in a production
 environment? If so, how large is your environment? Any thoughts of when
 Ceph will be production ready?



 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-deploy best practices

[John Wilkins]

Matthew,

I'll flag this for future doc changes. I noticed that you didn't run
ceph-deploy gatherkeys after creating your monitor(s). Any reason for that
omission?


On Mon, Dec 9, 2013 at 3:49 AM, Matthew Walster matt...@walster.org wrote:

 I'm having a play with ceph-deploy after some time away from it (mainly
 relying on the puppet modules).

 With a test setup of only two debian testing servers, I do the following:

 ceph-deploy new host1 host2
 ceph-deploy install host1 host2 (installs emperor)
 ceph-deploy mon create host1 host2
 ceph-deploy osd prepare host1:/dev/sda4 host2:/dev/sda4
 ceph-deploy osd activate host1:/dev/sda4 host2:/dev/sda4
 ceph-deploy mds create host1 host2

 Everything is running fine -- copy some files into CephFS, everything it
 looking great.

 host1: /etc/init.d/ceph stop osd

 Still fine.

 host1: /etc/init.d/ceph stop mds

 Fails over to the standby mds after a few seconds. Little outage, but to
 be expected. Everything fine.

 host1: /etc/init.d/ceph start osd
 host1: /etc/init.d/ceph start mds

 Everything recovers, everything is fine.

 Now, let's do something drastic:

 host1: reboot
 host2: reboot

 Both hosts come back up, but the mds never recovers -- it always says it
 is replaying.

 On closer inspection, host2's osd never came back into action. Doing:

 ceph-deploy osd activate host2:/dev/sda4 fixed the issue, and the mds
 recovered, as well as the osd now reporting both up and in.

 Is there something obvious I'm missing? The ceph.conf seemed remarkably
 empty, do I have to re-deploy the configuration file to the monitors or
 similar? I've never noticed a problem with puppet deployed hosts, but that
 manually writes out the ceph.conf as part of the puppet run.

 Many thanks in advance,

 Matthew Walster



 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Basic cephx configuration

[John Wilkins]

). Best wishes
 for the future of Ceph, and best regards.

 Nicolas Canceill
 Scalable Storage Systems
 SURFsara (Amsterdam, NL)


 [1] http://ceph.com/docs/master/rados/configuration/auth-
 config-ref/#keys



 On 11/29/2013 03:09 PM, nicolasc wrote:

 An update on this issue:

 Explicitly setting the keyring parameter to its default value, in the
 client section, like this:

 [client.admin]
 keyring = /etc/ceph/ceph.client.admin.keyring

 solves the problem in the particular case when ONLY
 auth_cluster_required is set to cephx, and the two remaining auth
 parameters are set to none.

 The documentation clearly states that /etc/ceph/ceph.client.admin.keyring
 is the default value of the keyring setting [1], so this looks like a
 bug. Should I report it on the tracker? (BTW, all of this is on v0.72.1.)

 Also, does anyone have any idea about why this is not enough to enable
 the auth_service_required setting? That one still gives me the error:

 client.admin authentication error (95) Operation not supported

 Best regards,

 Nicolas Canceill
 Scalable Storage Systems
 SURFsara (Amsterdam, NL)

 [1] http://ceph.com/docs/master/rados/configuration/auth-
 config-ref/#keys



 On 11/29/2013 10:22 AM, nicolasc wrote:

 Hello every one,

 Just ran a fresh install of version Emperor on an empty cluster, and I
 am left clueless, trying to troubleshoot cephx. After ceph-deploy created
 the keys, I used ceph-authtool to generate the client.admin keyring and 
 the
 monitor keyring, as indicated in the doc. The configuration is really
 out-of-the-box: 3 monitors, each with the keyring in
 /var/lib/ceph/mon/ceph-???/keyring, all keyrings have umask 644 and
 are owned by ceph.

 However, no matter which combination of auth_cluster_,
 auth_service_, or auth_client_required, is set to cephx; no matter
 either the keyring options like -k and --id on the command line.
 Authentication fails every time with:

 client.admin authentication error (95) Operation not supported
 Error connecting to cluster: Error

 A big thanks to any one who gives me a hint about what it means. (This
 message carries so little information, I feel it could be simply replaced
 by the ! character.) I have looked in every ceph and system log file,
 nothing more.

 Best regards,

 Nicolas Canceill
 Scalable Storage Systems
 SURFsara (Amsterdam, NL)

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] how to fix active+remapped pg

[John Wilkins]

Ugis,

Can you provide the results for:

ceph osd tree
ceph osd crush dump






On Thu, Nov 21, 2013 at 7:59 AM, Gregory Farnum g...@inktank.com wrote:
 On Thu, Nov 21, 2013 at 7:52 AM, Ugis ugi...@gmail.com wrote:
 Thanks, reread that section in docs and found tunables profile - nice
 to have, hadn't noticed it before(ceph docs develop so fast that you
 need RSS to follow all changes :) )

 Still problem persists in a different way.
 Did set profile optimal, reballancing started, but I had rbd
 delete in background, in the end cluster ended up with negative
 degradation %
 I think I have hit bug http://tracker.ceph.com/issues/3720   which is
 still open.
 I did restart osds one by one and negative degradation dissapeared.

 Afterwards I added extra ~900GB data, degradation growed in process to 0.071%
 This is rather http://tracker.ceph.com/issues/3747  which is closed,
 but seems to happen still.
 I did ceph osd out X; sleep 40; ceph osd in X for all osds,
 degradation % went away.

 In the end I still have 55 active+remapped pgs and no degradation %.
 pgmap v1853405: 2662 pgs: 2607 active+clean, 55 active+remapped; 5361
 GB data, 10743 GB used, 10852 GB / 21595 GB avail; 25230KB/s rd,
 203op/s

 I queried some of remapped pgs, do not see why they do not
 reballance(tunables are optimal now, checked).

 Where to look for the reason they are not reballancing? Is there
 something to look for in osd logs if debug level is increased?

 one of those:
 # ceph pg 4.5e query
 { state: active+remapped,
   epoch: 9165,
   up: [
 9],
   acting: [
 9,
 5],

 For some reason CRUSH is still failing to map all the PGs to two hosts
 (notice how the up set is only one OSD, so it's adding another one
 in acting) — what's your CRUSH map look like?
 -Greg
 Software Engineer #42 @ http://inktank.com | http://ceph.com
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] near full osd

[John Wilkins]

 shall not
 be entitled to read, disseminate, disclose or duplicate it.

 If you receive this e-mail unintentionally, please inform us immediately
 by reply and then delete it from your system. Although this information
 has been compiled with great care, neither IMC Financial Markets  Asset
 Management nor any of its related entities shall accept any responsibility
 for any errors, omissions or other inaccuracies in this information or for
 the consequences thereof, nor shall it be bound in any way by the contents
 of this e-mail or its attachments. In the event of incomplete or incorrect
 transmission, please return the e-mail to the sender and permanently delete
 this message and any attachments.

 Messages and attachments are scanned for all known viruses. Always scan
 attachments before opening them.



 --
 Software Engineer #42 @ http://inktank.com | http://ceph.com

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Manual Installation steps without ceph-deploy

[John Wilkins]

I've seen this before too. CentOS starts up without networking on by
default. In my case, the problem was that the monitors cannot form a
quorum and OSDs cannot find each other or monitors. Hence, you get
that broken pipe error. You either need to have the networking start
on startup before the OSDs, or start ceph after you boot up and ensure
the network is running properly. The nodes have to be able to reach
each other for Ceph to work. As for Ubuntu, I believe the networking
is on by default.

On Wed, Nov 6, 2013 at 1:35 PM, Trivedi, Narendra
narendra.triv...@savvis.com wrote:
 Hi All,



 I did a fresh install of Ceph (this might be like 10th or 11th install) on 4
 new VMs (one admin, one MON and two OSDs) built from CentOS 6.4 (x64) .iso ,
 did a yum update on all of them. They are all running on vmware ESXi 5.1.0.
 I did everything sage et al suggested (i.e. creation of /ceph/osd* and
 making sure /etc/ceph is present on all nodes. /etc/ceph gets created all
 the ceph-deploy install and contains rbdmap FYI). Unusually, I ended up with
 the same problem while activating OSDs (the last 4 lines keep going on and
 on forever):



 2013-11-06 14:37:39,626 [ceph_deploy.cli][INFO  ] Invoked (1.3):
 /usr/bin/ceph-deploy osd activate ceph-node2-osd0-centos-6-4:/ceph/osd0
 ceph-node3-osd1-centos-6-4:/ceph/osd1

 2013-11-06 14:37:39,627 [ceph_deploy.osd][DEBUG ] Activating cluster ceph
 disks ceph-node2-osd0-centos-6-4:/ceph/osd0:
 ceph-node3-osd1-centos-6-4:/ceph/osd1:

 2013-11-06 14:37:39,901 [ceph-node2-osd0-centos-6-4][DEBUG ] connected to
 host: ceph-node2-osd0-centos-6-4

 2013-11-06 14:37:39,902 [ceph-node2-osd0-centos-6-4][DEBUG ] detect platform
 information from remote host

 2013-11-06 14:37:39,917 [ceph-node2-osd0-centos-6-4][DEBUG ] detect machine
 type

 2013-11-06 14:37:39,925 [ceph_deploy.osd][INFO  ] Distro info: CentOS 6.4
 Final

 2013-11-06 14:37:39,925 [ceph_deploy.osd][DEBUG ] activating host
 ceph-node2-osd0-centos-6-4 disk /ceph/osd0

 2013-11-06 14:37:39,925 [ceph_deploy.osd][DEBUG ] will use init type:
 sysvinit

 2013-11-06 14:37:39,925 [ceph-node2-osd0-centos-6-4][INFO  ] Running
 command: sudo ceph-disk-activate --mark-init sysvinit --mount /ceph/osd0

 2013-11-06 14:37:40,145 [ceph-node2-osd0-centos-6-4][ERROR ] 2013-11-06
 14:37:41.075310 7fac2414c700  0 -- :/1029546  10.12.0.70:6789/0
 pipe(0x7fac20024480 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7fac200246e0).fault

 2013-11-06 14:37:43,167 [ceph-node2-osd0-centos-6-4][ERROR ] 2013-11-06
 14:37:44.071697 7fac1ebfd700  0 -- :/1029546  10.12.0.70:6789/0
 pipe(0x7fac14000c00 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7fac14000e60).fault

 2013-11-06 14:37:46,140 [ceph-node2-osd0-centos-6-4][ERROR ] 2013-11-06
 14:37:47.071938 7fac2414c700  0 -- :/1029546  10.12.0.70:6789/0
 pipe(0x7fac14003010 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7fac14003270).fault

 2013-11-06 14:37:50,165 [ceph-node2-osd0-centos-6-4][ERROR ] 2013-11-06
 14:37:51.071245 7fac1ebfd700  0 -- :/1029546  10.12.0.70:6789/0
 pipe(0x7fac14003a70 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7fac14003cd0).fault



 It might be bad luck but I want to try a manual installation without
 ceph-deploy because it seems  I am jinxed with ceph-deploy. Could anyone
 please forward me the steps. I am happy to share the ceph.log with anyone
 who would like to research on this error but I don’t a have clue.





 Thanks a lot!

 Narendra Trivedi | savviscloud




 This message contains information which may be confidential and/or
 privileged. Unless you are the intended recipient (or authorized to receive
 for the intended recipient), you may not read, use, copy or disclose to
 anyone the message or any information contained in the message. If you have
 received the message in error, please advise the sender by reply e-mail and
 delete the message and any attachment(s) thereto without retaining any
 copies.

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw questions

[John Wilkins]

For #2, I just wrote a document on setting up a federated
architecture. You can view it here:
http://ceph.com/docs/master/radosgw/federated-config/ This
functionality will be available in the Emperor release.

The use case I described involved two zones in a master region talking
to the same underlying Ceph Storage Cluster, but with different sets
of pools for each zone. You can certainly set up pools for zones on
completely different Ceph Storage Clusters. I assumed that was
overkill, but you can certainly do it. See
http://ceph.com/docs/master/radosgw/federated-config/#configure-a-master-region
for configuring a master region.

If you want to use separate storage clusters for each zone, you need to:

1. Setup the set of pools for each zone in the respective ceph storage
cluster for your data center.
2. http://ceph.com/docs/master/radosgw/federated-config/#create-a-keyring
should use different cluster names to ensure that the keyring gets
populated in both Ceph Storage Clusters. We assume the default -c
/etc/ceph/ceph.conf for simplicity.
3. 
http://ceph.com/docs/master/radosgw/federated-config/#add-instances-to-ceph-config-file
when adding the instances to the Ceph configuration file, you need to
note that the storage cluster might be named. For example, instead of
ceph.conf, it might be us-west.conf and us-east.conf for the
respective zones, assuming you are setting up Ceph clusters
specifically to run the gateways--or whatever naming convention you
already use.

4. Most of the usage examples omit the Ceph configuration file (-c
file/path.conf) and the admin key (-k path/to/admin.keyring). You may
need to specify them explicitly when calling radosgw-admin so that you
are issuing commands to the right Ceph Storage Cluster.

I'd love to get your feedback on the document!

For #3. Yes. In fact, if you just setup a master region with one
master zone, that works fine. You don't have to respect pool naming.
Whatever you create in the storage cluster and map to a zone pool will
work. However, I would suggest following the conventions as laid out
in the document. You can create a garbage collection pool called
lemonade, but you will probably confuse the community when looking
for help as they will expect .{region-name}-{zone-name}.rgw.gc. If you
just use region-zone.{pool-name-default}, like us-west.rgw.root most
people in the community will understand any questions you have and can
more readily help you with additional questions.




On Wed, Nov 6, 2013 at 3:17 AM, Alessandro Brega
alessandro.bre...@gmail.com wrote:
 Good day ceph users,

 I'm new to ceph but installation went well so far. Now I have a lot of
 questions regarding radosgw. Hope you don't mind...

 1. To build a high performance yet cheap radosgw storage, which pools should
 be placed on ssd and which on hdd backed pools? Upon installation of
 radosgw, it created the following pools: .rgw, .rgw.buckets,
 .rgw.buckets.index, .rgw.control, .rgw.gc, .rgw.root, .usage, .users,
 .users.email.

 2. In order to have very high availability I like to setup two different
 ceph clusters, each in its own datacenter. How to configure radowsgw to make
 use of this layout? Can I have a multi-master setup with having a load
 balancer (or using geo-dns) which distributes the load to radosgw instances
 in both datacenters?

 3. Is it possible to start with a simple setup now (only one ceph cluster)
 and later add the multi-datacenter redundancy described above without
 downtime? Do I have to respect any special pool-naming requirements?

 4. Which number of replaction would you suggest? In other words, which
 replication is need to achive 99.9% durability like dreamobjects states?

 5. Is it possible to map fqdn custom domain to buckets, not only subdomains?

 6. The command radosgw-admin pool list returns could not list placement
 set: (2) No such file or directory. But radosgw seems to work as expected
 anyway?

 Looking forward to your suggestions.

 Alessandro Brega


 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CONFIGURING FEDERATED GATEWAYS

[John Wilkins]

It will be updated by the end of the day today...

On Sun, Oct 27, 2013 at 7:31 PM, maoqi1982 maoqi1...@126.com wrote:
 Hi list
 my ceph version is dumpling 0.67 ,i want use RGW Geo-Replication and
 Disaster Recovery function, can I refer the doc
 http://ceph.com/docs/wip-doc-radosgw/radosgw/federated-config/ ( v0.71 )  to
 deploy the region /zones/agent



 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw-agent 403 error

[John Wilkins]

That looks like a permissions problem.  I've updated the draft
document here: http://ceph.com/docs/wip-doc-radosgw/radosgw/federated-config/



On Mon, Oct 28, 2013 at 2:25 AM, lixuehui lixue...@chinacloud.com.cn wrote:
 Hi all
 Today I'd like to replicated one cluster with gateway.After  master zone and
 slave zone working , i started a radosgw-agent .Unfortuntly , the agent
 return 403 error all the time .
 This is the master zone's information:

 Mon, 28 Oct 2013 09:19:29 GMT
 /admin/log
 2013-10-28 17:19:29.397742 7f7b55fdb700 15 calculated
 digest=PDQrUcKw0Tut6orsyhrg2sGBvG8=
 2013-10-28 17:19:29.397760 7f7b55fdb700 15
 auth_sign=PDQrUcKw0Tut6orsyhrg2sGBvG8=
 2013-10-28 17:19:29.397770 7f7b55fdb700 15 compare=0
 2013-10-28 17:19:29.397792 7f7b55fdb700  2 req 87:0.001841::GET
 /admin/log:get_metadata_log_info:reading permissions
 2013-10-28 17:19:29.397818 7f7b55fdb700  2 req 87:0.001868::GET
 /admin/log:get_metadata_log_info:verifying op mask
 2013-10-28 17:19:29.397839 7f7b55fdb700 20 required_mask= 0 user.op_mask=7
 2013-10-28 17:19:29.397852 7f7b55fdb700  2 req 87:0.001902::GET
 /admin/log:get_metadata_log_info:verifying op permissions
 2013-10-28 17:19:29.397988 7f7b55fdb700  2 req 87:0.002037::GET
 /admin/log:get_metadata_log_info:http status=403
 2013-10-28 17:19:29.398393 7f7b55fdb700  1 == req done req=0x1ce0770
 http_status=403 ==

 the agent command is:

  radosgw-agent--src-access-key BDKNCW56I2JWVD5OKD99  --src-secret-key
 O7IIFHwRw2EsG5n9Qjx4POquMjT0MlU7PSoTjiXq   --src-host 192.168.50.105
 --src-port 80  --src-zone ex0-0  --dest-access-key  YFP0CA81IZ0KJUMII09Z
 --dest-secret-key  vI0dYFycGHFFB5yO4eJuAjmJCxDN7j5NRzT3cRQs   --dest-host
 192.168.50.92  --dest-port 80  --dest-zone ex0-1  --daemon-id backup1


 infact the master zone user own permission like this

 { user_id: ex0-0,
   display_name: region0 zone0,
   email: ,
   suspended: 0,
   max_buckets: 1000,
   auid: 0,
   subusers: [],
   keys: [
 { user: ex0-0,
   access_key: BDKNCW56I2JWVD5OKD99,
   secret_key: O7IIFHwRw2EsG5n9Qjx4POquMjT0MlU7PSoTjiXq}],
   swift_keys: [],
   caps: [
 { type: usage,
   perm: *}],
   op_mask: read, write, delete,
   default_placement: ,
   placement_tags: []}

 I wander where's the problem ,thanks for any help!
 
 lixuehui

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CONFIGURING FEDERATED GATEWAYS

[John Wilkins]

I still need to update the graphics.  The update text is here:
http://ceph.com/docs/wip-doc-radosgw/radosgw/federated-config/

On Mon, Oct 28, 2013 at 11:49 AM, John Wilkins john.wilk...@inktank.com wrote:
 It will be updated by the end of the day today...

 On Sun, Oct 27, 2013 at 7:31 PM, maoqi1982 maoqi1...@126.com wrote:
 Hi list
 my ceph version is dumpling 0.67 ,i want use RGW Geo-Replication and
 Disaster Recovery function, can I refer the doc
 http://ceph.com/docs/wip-doc-radosgw/radosgw/federated-config/ ( v0.71 )  to
 deploy the region /zones/agent



 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




 --
 John Wilkins
 Senior Technical Writer
 Intank
 john.wilk...@inktank.com
 (415) 425-9599
 http://inktank.com



-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph + Xen - RBD io hang

[John Wilkins]

John,

I've never installed anything on Scientific Linux. Are you sure that
QEMU has RBD support?

I have some wip-doc text, which I'm going to move around shortly. You
can see the yum install requirements here:

http://ceph.com/docs/wip-doc-install/install/yum-priorities/
http://ceph.com/docs/wip-doc-install/install/install-qemu/#rpm-packages

Make sure you can run QEMU commands on RBD first, since that's a dependency.

http://ceph.com/docs/wip-doc-install/rbd/qemu-rbd/

On Mon, Oct 28, 2013 at 12:01 PM, John Morris j...@zultron.com wrote:
 I'm encountering a problem with RBD-backed Xen.  During a VM boot,
 pygrub attaches the VM's root VDI to dom0.  This hangs with these
 messages in the debug log:

 Oct 27 21:19:59 xen27 kernel:
   vbd vbd-51728: 16 Device in use; refusing to close
 Oct 27 21:19:59 xen27 xenopsd-xenlight:
   [xenops] waiting for backend to close
 Oct 27 21:19:59 xen27 kernel:
   qemu-system-i38[2899]: segfault at 7fac042e4000 ip 7fac0447b129
   sp 7fffe7028630 error 4 in qemu-system-i386[7fac042ed000+309000]

 More details here:

 http://pastebin.ca/2472234

   - Scientific Linux 6
   - 64-bit, Phenom CPU
   - Ceph from RPM ceph-0.67.4-0.el6.x86_64
   - XenAPI from Dave Scott's technology preview
   - two btrfs-backed OSDs with journals on separate drives
   - various kernels, incl. 3.4.6 from Dave Scott's repo and 3.11.6
 from elrepo.org.

 This thread (whose Subject: I borrowed) describes what I'm seeing quite
 well, but no resolution was posted:

 http://comments.gmane.org/gmane.comp.file-systems.ceph.user/3636

 In my case, udevd starts a 'blkid' process that holds /dev/xvdb open.
 Like in James's case, any interaction with the device will hang, and
 usually can't be killed.  This same problem prevents the machine from
 completing shutdown.

 In that thread, Sylvain Munaut says the OSD and kernel driver shouldn't
 be run in the same host.  I believe my setup does not violate that,
 since the rbd kernel module is not loaded, and instead the device is
 attached through the xen_blkfront module instead.

 Thanks-

 John
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Install Guide - CEPH WITH OPENSTACK

[John Wilkins]

Raghavendra,

You can follow the link Loic provided. If you are running on
CentOS/RHEL, make sure you install QEMU with RBD support. See
http://ceph.com/docs/master/install/qemu-rpm/

Make sure your QEMU and libvirt installs are working. Then do the
integration with OpenStack.

On Mon, Oct 28, 2013 at 1:39 AM, Loic Dachary l...@dachary.org wrote:
 Hi,

 Is this what you're looking for : 
 http://ceph.com/docs/next/rbd/rbd-openstack/ ?

 Cheers

 On 28/10/2013 04:05, Raghavendra Lad wrote:



 Hi Cephs,

 I am new to Ceph. I am planning to install CEPH.

 I already have Openstack Grizzly installed and for storage thought of 
 installing Ceph.

 Please could some one help us with the Tutorial / Install Guide to install 
 CEPH TO INTEGRATE WITH OPENSTACK.

 Regards,

 Raghavendra Lad



 http://sigads.rediff.com/RealMedia/ads/click_nx.ads/www.rediffmail.com/signatureline.htm@Middle?

 Get your own *FREE* website, *FREE* domain  *FREE* mobile app with Company 
 email.
   *Know More * 
 http://track.rediff.com/click?url=___http://businessemail.rediff.com/company-email-hosting-services?sc_cid=sign-1-10-13___cmp=hostlnk=sign-1-10-13nsrv1=host



 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


 --
 Loïc Dachary, Artisan Logiciel Libre
 All that is necessary for the triumph of evil is that good people do nothing.


 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Fwd: FW: ceph-deploy again

[John Wilkins]

The original documentation was written with a script called mkcephfs
in mind. Then, we began including some documentation for Chef and
Crowbar.  We actually only had developer documentation for doing
things manually. We're working on providing manual steps now. While
it's not in the deployment section, there are some docs that may help
you understand what's going on when you deploy manually. First you
have to bootstrap a monitor. So we have some developer documentation
on that here:

http://ceph.com/docs/master/dev/mon-bootstrap/

There are also some docs for manually adding monitors and OSDs to an
existing cluster here:

http://ceph.com/docs/master/rados/operations/add-or-rm-mons/
http://ceph.com/docs/master/rados/operations/add-or-rm-osds/

There are also some documents for creating and modifying CRUSH maps here:

http://ceph.com/docs/master/rados/operations/crush-map/

There's a high level discussion of why we use the maps here:

http://ceph.com/docs/master/architecture/#scalability-and-high-availability

These procedural docs need some updating too, and we should have a
manual deploy section up soon.





John

On Sun, Oct 6, 2013 at 8:45 PM, Harvey Skinner hpmpe...@gmail.com wrote:
 hi John, thank you for the reply on this.

 From further experimentation, it is the ceph-deploy v1.2.6 package for
 Cuttlefish that is broken or incorrectly defined/packaged.  I cleaned
 up the broken packages and tried again multiple times with same
 results.  You can install ceph-deploy from the ceph repo for
 Cuttlefish still, but as mentioned previously it is only v1.0-1 and
 full of known issues.

 For fun I reset my repo links for:
 deb http://ceph.com/debian-dumpling/ raring main
 deb http://ceph.com/packages/ceph-extras/debian/ raring main

 to check the dumpling version of ceph-deploy ... and it installed
 without problem.  The differece being that the ceph-deploy package in
 the ceph-extras has a specific python-pushy package called
 'python-pushy_0.5.3-1raring.ceph_amd64.deb', apparently put together
 by ceph folks for deployment with ceph-deploy v1.2.6 w/ Dumpling 
 Raring.The problem is the same was not done for Cuttlefish 
 Raring.   So trying to install ceph-deploy for use with Cuttlefish(?)
 or at least on Raring is a waste of time.

 So even though I had not planned to, I am going with Dumpling release.
   But I have ran into additional problems with ceph-deploy v1.2.6 and
 cannot use 'ceph-deploy install' to install ceph on my ceph nodes.
 It tries to install the ceph release key as one of the first steps and
 hangs.   From further experimentation, it is hanging because the
 command to retrieve the release key and add it is missing a 'sudo'
 command in it for the 'apt-key add' part of the command:

 Running command: wget -q -O-
 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' |
 apt-key add –

 When you reissue the above command with a 'sudo' preceding the
 'apt-key add -' it works fine.   So I ended up installing ceph bits on
 all my ceph nodes by hand.

 I could now create a 'new' cluster with ceph-deploy and thought I
 would try using a cluster name different than the default of 'ceph' to
 see how that works, I saw an e-mail saying you can have multiple
 clusters on same h/w and wanted to try it.  But it seems that
 ceph-deploy doesn't like working with clusters that aren't using the
 defalt cluster name of 'ceph'.  When I tried to add MONs it couldn't
 find 'ceph.conf' because it was named 'cluster-name.conf' and
 errored out.

 SO there are still issues with ceph-deploy that cause a lot of
 investigation and workaround time  effort and still have to accept
 its limitations.   The bad part of this is all the current online
 documentation is being changed to reference use of ceph-deploy only
 and any documentation of how to do tasks without ceph-deploy (to get
 around ceph-deploy limitations) is missing.

 Harvey

 On Thu, Oct 3, 2013 at 10:17 AM, John Wilkins john.wilk...@inktank.com 
 wrote:
 I've run into this before too. I think with broken packages, you have to
 uninstall the previous version and do apt-get autoremove as well. Sometimes
 you have to manually uninstall whatever it lists as the broken packages and
 then do autoremove. Then, reinstall.


 On Thu, Oct 3, 2013 at 12:20 AM, Harvey Skinner hpmpe...@gmail.com wrote:

 I am also having problems getting the latest version of ceph-deploy to
 install on Raring.
 I was able to install the updated ceph-deploy about two months ago for
 Ubuntu 12.04 and Cuttlefish using the following two lines in the
 '/etc/apt/sources.list.d/ceph.list'  apt sources file on my 'Admin' node:

deb http://ceph.com/debian-cuttlefish/ precise main
deb http://ceph.com/packages/ceph-extras/debian/ precise main

 This worked to download the latest version of ceph-deploy, I believe it
 was 1.2-0 or something like that.

  I have now re-imaged my servers to Ubuntu 13.04 and going through the
 steps to deploy a new Ceph cluster using

Re: [ceph-users] Fwd: FW: ceph-deploy again

[John Wilkins]

 Institut gemeinnützige GmbH | Pfalzburger Str. 43/44 | 10717
 Berlin | Germany
 GF: R. Andreas Kraemer | AG: Charlottenburg HRB 57947 | USt/VAT-IdNr.:
 DE811963464
 Ecologic™ is a Trade Mark (TM) of Ecologic Institut gemeinnützige GmbH ***
 *

   --

 ___
 ceph-users mailing list
 *ceph-users@lists.ceph.com*
 *http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com*

 ** **



 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] trouble adding OSDs - which documentation to use

[John Wilkins]

Jogi,

I'm working on updating for manual installation. Most users who want
that level of detail are using it to incorporate ceph into another
deployment system like Chef, Puppet, Juju, etc. What you are working
on is pre ceph-deploy documentation, and some chef commentary too.
The links are broken for chef. I'll fix those. If you initially
deployed your cluster with ceph-deploy, you are probably best off
using ceph-deploy documentation here:
http://ceph.com/docs/master/rados/deployment/

On Tue, Oct 1, 2013 at 8:08 AM, Jogi Hofmüller j...@mur.at wrote:
 Dear all,

 I am back to managing the cluster before starting to use it even on a
 test host.  First of all a question regarding the docs:

 Is this [1] outdated?  If not, why are the links to chef-* not working?
  Is chef-* still recommended/used?

 After adding a new OSD (with ceph-deploy version 1.2.6) and starting the
 daemon after a reboot of the osd-server it complains:

 root@ceph-server1:~# service ceph start
 === osd.0 ===
 No filesystem type defined!

 I could not find anything in the docs on how to specify the fs-type.
 How is mounting the data-partition done usually?  It works if I mount it
 via an entry in /etc/fstab (or manually) but I would have to edit that
 manually.

 All this is done using ceph dumpling installed/deployed according to
 the getting started info from [2].

 [1]  http://ceph.com/docs/master/rados/operations/add-or-rm-osds/
 [2]  http://ceph.com/docs/master/start/quick-ceph-deploy/

 Regards!
 --
 j.hofmüller

 Optimism doesn't alter the laws of physics. - Subcommander T'Pol


 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph.conf changes and restarting ceph.

[John Wilkins]

From your pastie details, it looks like you are using auth supported
= none.  That's pre 0.51, as noted in the documentation. Perhaps I
should omit the old usage or omit it entirely.

It should look like this:

auth cluster required = none
auth service required = none
auth client required = none

not

auth supported = none

On Tue, Sep 24, 2013 at 8:00 AM, Aronesty, Erik
earone...@expressionanalysis.com wrote:
 I did the same thing, restarted with upstart, and I still need to use
 authentication.   Not sure why yet.   Maybe I didn’t change the /etc/ceph
 configs on all the nodes….



 From: ceph-users-boun...@lists.ceph.com
 [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Snider, Tim
 Sent: Tuesday, September 24, 2013 9:15 AM
 To: Gary Mazzaferro; John Wilkins
 Cc: ceph-users@lists.ceph.com


 Subject: Re: [ceph-users] ceph.conf changes and restarting ceph.



 Authentication works. I was interested in trying it without authentication.
 I didn’t see the upstart link earlier.

 Is the plan to only use upstart and not service for Dumpling and beyond?

 Tim



 From: Gary Mazzaferro [mailto:ga...@oedata.com]
 Sent: Tuesday, September 24, 2013 1:16 AM
 To: John Wilkins
 Cc: Snider, Tim; ceph-users@lists.ceph.com
 Subject: Re: [ceph-users] ceph.conf changes and restarting ceph.



 Hi John



 Why ? do the 'service' scripts not work ? (sorry I don't have access to the
 systems from my location) I used dumpling and ceph-deploy on debian.



 -gary









 On Mon, Sep 23, 2013 at 11:25 PM, John Wilkins john.wilk...@inktank.com
 wrote:

 I will update the Cephx docs. The usage in those docs for restarting
 is for Debian/Ubuntu deployed with mkcephfs.  If you are using
 Dumpling and deployed with ceph-deploy, you will need to use Upstart.
 See
 http://ceph.com/docs/master/rados/operations/operating/#running-ceph-with-upstart
 for details. If you are using Ceph on RHEL, CentOS, etc., use
 sysvinit.


 On Mon, Sep 23, 2013 at 3:21 PM, Gary Mazzaferro ga...@oedata.com wrote:
 Tim

 Did it work with authentication enabled  ?

 -gary


 On Mon, Sep 23, 2013 at 2:10 PM, Snider, Tim tim.sni...@netapp.com
 wrote:

 I modified /etc/ceph.conf for no authentication and to specify both
 private and public networks. /etc/ceph/ceph.conf was distributed to all
 nodes in the cluster

 ceph was restarted on all nodes using  service ceph -a restart.

 After that authentication is still required and no ports are open on the
 cluster facing (192.168.10.0) network.

 Details in  http://pastie.org/8349534.

 What am I missing something?



 Thanks,

 Tim


 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


 --
 John Wilkins
 Senior Technical Writer
 Intank
 john.wilk...@inktank.com
 (415) 425-9599
 http://inktank.com





-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph.conf changes and restarting ceph.

[John Wilkins]

Either one should work. For RHEL, CentOS, etc., use sysvinit.

I rewrote the ops doc, but it's in a wip branch right now. Here:
http://ceph.com/docs/wip-doc-quickstart/rados/operations/operating/

I still may make some edits to it, but follow the sysvinit section.


On Tue, Sep 24, 2013 at 10:08 AM, Snider, Tim tim.sni...@netapp.com wrote:
 Is the form: auth cluster required = none or auth_cluster_required = none? 
 (_s as a word separator)

 -Original Message-
 From: John Wilkins [mailto:john.wilk...@inktank.com]
 Sent: Tuesday, September 24, 2013 11:43 AM
 To: Aronesty, Erik
 Cc: Snider, Tim; Gary Mazzaferro; ceph-users@lists.ceph.com
 Subject: Re: [ceph-users] ceph.conf changes and restarting ceph.

 From your pastie details, it looks like you are using auth supported = 
 none.  That's pre 0.51, as noted in the documentation. Perhaps I should omit 
 the old usage or omit it entirely.

 It should look like this:

 auth cluster required = none
 auth service required = none
 auth client required = none

 not

 auth supported = none

 On Tue, Sep 24, 2013 at 8:00 AM, Aronesty, Erik 
 earone...@expressionanalysis.com wrote:
 I did the same thing, restarted with upstart, and I still need to use
 authentication.   Not sure why yet.   Maybe I didn’t change the /etc/ceph
 configs on all the nodes….



 From: ceph-users-boun...@lists.ceph.com
 [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Snider, Tim
 Sent: Tuesday, September 24, 2013 9:15 AM
 To: Gary Mazzaferro; John Wilkins
 Cc: ceph-users@lists.ceph.com


 Subject: Re: [ceph-users] ceph.conf changes and restarting ceph.



 Authentication works. I was interested in trying it without authentication.
 I didn’t see the upstart link earlier.

 Is the plan to only use upstart and not service for Dumpling and beyond?

 Tim



 From: Gary Mazzaferro [mailto:ga...@oedata.com]
 Sent: Tuesday, September 24, 2013 1:16 AM
 To: John Wilkins
 Cc: Snider, Tim; ceph-users@lists.ceph.com
 Subject: Re: [ceph-users] ceph.conf changes and restarting ceph.



 Hi John



 Why ? do the 'service' scripts not work ? (sorry I don't have access
 to the systems from my location) I used dumpling and ceph-deploy on debian.



 -gary









 On Mon, Sep 23, 2013 at 11:25 PM, John Wilkins
 john.wilk...@inktank.com
 wrote:

 I will update the Cephx docs. The usage in those docs for restarting
 is for Debian/Ubuntu deployed with mkcephfs.  If you are using
 Dumpling and deployed with ceph-deploy, you will need to use Upstart.
 See
 http://ceph.com/docs/master/rados/operations/operating/#running-ceph-w
 ith-upstart for details. If you are using Ceph on RHEL, CentOS, etc.,
 use sysvinit.


 On Mon, Sep 23, 2013 at 3:21 PM, Gary Mazzaferro ga...@oedata.com wrote:
 Tim

 Did it work with authentication enabled  ?

 -gary


 On Mon, Sep 23, 2013 at 2:10 PM, Snider, Tim tim.sni...@netapp.com
 wrote:

 I modified /etc/ceph.conf for no authentication and to specify both
 private and public networks. /etc/ceph/ceph.conf was distributed to
 all nodes in the cluster

 ceph was restarted on all nodes using  service ceph -a restart.

 After that authentication is still required and no ports are open on
 the cluster facing (192.168.10.0) network.

 Details in  http://pastie.org/8349534.

 What am I missing something?



 Thanks,

 Tim


 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


 --
 John Wilkins
 Senior Technical Writer
 Intank
 john.wilk...@inktank.com
 (415) 425-9599
 http://inktank.com





 --
 John Wilkins
 Senior Technical Writer
 Intank
 john.wilk...@inktank.com
 (415) 425-9599
 http://inktank.com



-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] clients in cluster network?

[John Wilkins]

Clients use the public network. The cluster network is principally for
OSD-to-OSD communication--heartbeats, replication, backfill, etc.

On Mon, Sep 23, 2013 at 7:42 AM, Kurt Bauer kurt.ba...@univie.ac.at wrote:
 Hi,
  just a short question to which I couldn't find an answer in the
 documentation:
 When I run a cluster with public and cluster network seperated, would it
 still be possible to have clients accessing the cluster (ie. RBDs) from
 within the cluster network?

 Thanks for your help,
 best regards,
 Kurt


 --
 Kurt Bauer kurt.ba...@univie.ac.at
 Vienna University Computer Center - ACOnet - VIX
 Universitaetsstrasse 7, A-1010 Vienna, Austria, Europe
 Tel: ++43 1 4277 - 14070 (Fax: - 9140)  KB1970-RIPE

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph.conf changes and restarting ceph.

[John Wilkins]

I will update the Cephx docs. The usage in those docs for restarting
is for Debian/Ubuntu deployed with mkcephfs.  If you are using
Dumpling and deployed with ceph-deploy, you will need to use Upstart.
See 
http://ceph.com/docs/master/rados/operations/operating/#running-ceph-with-upstart
for details. If you are using Ceph on RHEL, CentOS, etc., use
sysvinit.

On Mon, Sep 23, 2013 at 3:21 PM, Gary Mazzaferro ga...@oedata.com wrote:
 Tim

 Did it work with authentication enabled  ?

 -gary


 On Mon, Sep 23, 2013 at 2:10 PM, Snider, Tim tim.sni...@netapp.com wrote:

 I modified /etc/ceph.conf for no authentication and to specify both
 private and public networks. /etc/ceph/ceph.conf was distributed to all
 nodes in the cluster

 ceph was restarted on all nodes using  service ceph -a restart.

 After that authentication is still required and no ports are open on the
 cluster facing (192.168.10.0) network.

 Details in  http://pastie.org/8349534.

 What am I missing something?



 Thanks,

 Tim


 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help with radosGW

[John Wilkins]

I see that you added your public and cluster networks under an [osd]
section. All daemons use the public network, and OSDs use the cluster
network. Consider moving those settings to [global].
http://ceph.com/docs/master/rados/configuration/network-config-ref/#ceph-networks

Also, I do believe I had a doc bug to fix here.
http://tracker.ceph.com/issues/6182  It is now resolved. The s3gw.fcgi
file should be in /var/www as suggested. However, my chmod instruction
pointed to an incorrect directory. Can you take a look at that and see
if that helps?

On Tue, Sep 17, 2013 at 1:29 AM, Alexis GÜNST HORN
alexis.gunsth...@outscale.com wrote:
 Hello to all,

 I've a big issue with Ceph RadosGW.
 I did a PoC some days ago with radosgw. It worked well.

 Ceph version 0.67.3 under CentOS 6.4

 Now, I'm installing a new cluster but, I can't succeed. I do not understand 
 why.
 Here is some elements :

 ceph.conf:

 [global]
 filestore_xattr_use_omap = true
 mon_host = 192.168.0.1,192.168.0.2,192.168.0.3
 fsid = f261d4c5-2a93-43dc-85a9-85211ec7100f
 mon_initial_members = mon-1, mon-2, mon-3
 auth_supported = cephx
 osd_journal_size = 10240

 [osd]
 cluster_network = 192.168.0.0/24
 public_network = 192.168.1.0/24


 [client.radosgw.gateway]
 host = gw-1
 keyring = /etc/ceph/keyring.radosgw.gateway
 rgw socket path = /tmp/radosgw.sock
 log file = /var/log/ceph/radosgw.log
 rgw print continue = false



 I followed this doc to install radosgw :
 http://ceph.com/docs/next/install/rpm/#installing-ceph-object-storage

 I start httpd :
 /etc/init.d/httpd start

 I start radosgw :
 [root@gw-1]# /etc/init.d/ceph-radosgw start
 Starting radosgw instance(s)...
 2013-09-17 08:07:11.954248 7f835d7fb820 -1 WARNING: libcurl doesn't
 support curl_multi_wait()
 2013-09-17 08:07:11.954253 7f835d7fb820 -1 WARNING: cross zone /
 region transfer performance may be affected

 I create a user :
 radosgw-admin user create --uid=alexis

 It works.
 Fine.

 So now, I connect to the gateway via a client (CyberDuck).
 I can create a bucket : test.
 Then, I try to upload a file = does not work.
 I have a time out after about 30 secs.

 And, of course, the file is not uploaded. A rados df on .rgw.buckets
 show that there is no objects inside.

 Here are some logs.

 radosgw.log:
 http://pastebin.com/6NNuczC5
 (the last lines are because I stop radosgw, not to pollute the logs)

 and httpd.log :
 [Tue Sep 17 08:02:15 2013] [error] [client 46.231.147.8] FastCGI: comm
 with server /var/www/s3gw.fcgi aborted: idle timeout (30 sec)
 [Tue Sep 17 08:02:15 2013] [error] [client 46.231.147.8] FastCGI:
 incomplete headers (0 bytes) received from server /var/www/s3gw.fcgi
 [Tue Sep 17 08:02:45 2013] [error] [client 46.231.147.8] FastCGI: comm
 with server /var/www/s3gw.fcgi aborted: idle timeout (30 sec)
 [Tue Sep 17 08:02:45 2013] [error] [client 46.231.147.8] FastCGI:
 incomplete headers (0 bytes) received from server /var/www/s3gw.fcgi
 [Tue Sep 17 08:08:42 2013] [error] [client 46.231.147.8] FastCGI: comm
 with server /var/www/s3gw.fcgi aborted: idle timeout (30 sec)
 [Tue Sep 17 08:08:46 2013] [error] [client 46.231.147.8] FastCGI:
 incomplete headers (0 bytes) received from server /var/www/s3gw.fcgi
 [Tue Sep 17 08:12:35 2013] [error] [client 46.231.147.8] FastCGI: comm
 with server /var/www/s3gw.fcgi aborted: idle timeout (30 sec)
 [Tue Sep 17 08:12:35 2013] [error] [client 46.231.147.8] FastCGI:
 incomplete headers (0 bytes) received from server /var/www/s3gw.fcgi
 [Tue Sep 17 08:13:02 2013] [error] [client 46.231.147.8] FastCGI:
 incomplete headers (0 bytes) received from server /var/www/s3gw.fcgi



 I'm really diapointed because i can't understand where is the issue.
 Thanks A LOT for your help.

 Alexis
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] getting started

[John Wilkins]

We will have a new update to the quick start this week.

On Mon, Sep 16, 2013 at 12:18 PM, Alfredo Deza alfredo.d...@inktank.com wrote:
 On Mon, Sep 16, 2013 at 12:58 PM, Justin Ryan justin.r...@kixeye.com wrote:
 Hi,

 I'm brand new to Ceph, attempting to follow the Getting Started guide with 2
 VMs. I completed the Preflight without issue.  I completed Storage Cluster
 Quick Start, but have some questions:

 The Single Node Quick Start grey box -- does 'single node' mean if you're
 running the whole thing on a single machine, if you have only one server
 node like the diagram at the top of the page, or if you're only running one
 OSD process? I'm not sure if I need to make the `osd crush chooseleaf type`
 change.

 Are the LIST, ZAP, and ADD OSDS ON STANDALONE DISKS sections an alternative
 to the MULTIPLE OSDS ON THE OS DISK (DEMO ONLY) section? I thought I set up
 my OSDs already on /tmp/osd{0,1}.

 Moving on to the Block Device Quick Start -- it says To use this guide, you
 must have executed the procedures in the Object Store Quick Start guide
 first -- but the link to the Object Store Quick Start actually points to
 the Storage Cluster Quick Start -- which is it?

 Most importantly, it says Ensure your Ceph Storage Cluster is in an active
 + clean state before working with the Ceph Block Device --- how can tell if
 my cluster is active+clean?? The only ceph* command on the admin node is
 ceph-deploy, and running `ceph` on the server node:

 ceph@jr-ceph2:~$ ceph
 2013-09-16 16:53:10.880267 7feb96c1b700 -1 monclient(hunting): ERROR:
 missing keyring, cannot use cephx for authentication
 2013-09-16 16:53:10.880271 7feb96c1b700  0 librados: client.admin
 initialization error (2) No such file or directory
 Error connecting to cluster: ObjectNotFound

 There is a ticket open for this, but you basically need super-user
 permissions here to run (any?) ceph commands.

 Thanks in advance for any help, and apologies if I missed anything obvious.





 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] problem with ceph user

[John Wilkins]

Maybe a doc bug somewhere?  Quick start preflight says, wget -q -O-
'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' |
sudo apt-key add -

You need to have sudo before apt-key add -

On Fri, Sep 13, 2013 at 1:16 PM, Gruher, Joseph R
joseph.r.gru...@intel.com wrote:
 Hello all-



 I’m setting up a new Ceph cluster (my first time – just a lab experiment,
 not for production) by following the docs on the ceph.com website.  The
 preflight checklist went fine, I installed and updated Ubuntu 12.04.2, set
 up my user and set up passwordless SSH, etc.  I ran “ceph-deploy new”
 without any apparent issues.  However, when I run “ceph-deploy install” it
 hangs at this point:



 [cephtest02][INFO  ] Running command: wget -q -O-
 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' | apt-key
 add -



 It looks to me like it is failing on the “apt-key add” command.  If I log
 directly into the cephtest02 host as my ceph user and try to run “apt-key
 add” it fails:



 $ apt-key add

 ERROR: This command can only be used by root.



 It works if I include a “sudo”:



 $ sudo apt-key add

 gpg: can't open `': No such file or directory



 So I assume the problem is my ceph user doesn’t have the right permissions?
 I set up the ceph user by following the instructions in the preflight
 checklist (http://ceph.com/docs/master/start/quick-start-preflight/):



 root@cephtest02:/# cat /etc/sudoers.d/ceph

 ceph ALL = (root) NOPASSWD:ALL



 root@cephtest02:/# ls -l /etc/sudoers.d/ceph

 -r--r- 1 root root 31 Sep 12 15:45 /etc/sudoers.d/ceph



 $ sudo -l

 Matching Defaults entries for ceph on this host:

 env_reset,


 secure_path=/usr/local/sbin\:/usr/local/bin\:/usr/sbin\:/usr/bin\:/sbin\:/bin



 User ceph may run the following commands on this host:

 (root) NOPASSWD: ALL



 Can anyone tell me where I’m going wrong here, or in general how to give the
 ceph user the appropriate permissions?  Or is this a ceph-deploy problem
 that it is not including the “sudo”?



 Thanks,

 Joe


 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rgw geo-replication and disaster recovery problem

[John Wilkins]

I'm working on a draft document of how to set up regions and zones
with metadata replication. Data replication is on the way, but I
haven't worked with it yet. Let me know how it goes, because this
still requires some testing and user feedback.

http://ceph.com/docs/wip-doc-radosgw/radosgw/federated-config/

On Mon, Sep 9, 2013 at 1:50 PM, Samuel Just sam.j...@inktank.com wrote:
 The regions and zones can be used to distribute among different ceph clusters.
 -Sam

 On Mon, Sep 2, 2013 at 2:05 AM, 李学慧 lixuehui...@126.com wrote:
 Mr.
 Hi!I'm interested into the rgw geo-replication and disaster recovery
 feature.
 But whether those 'regisions and zones ' distributes among several different
 ceph clusters or just only one?
 Thank you !


 

 ashely



 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Documentation OS Recommendations

[John Wilkins]

Yes. We'll have an update shortly.

On Mon, Sep 9, 2013 at 11:29 AM, Scottix scot...@gmail.com wrote:
 I was looking at someones question on the list and started looking up some
 documentation and found this page.
 http://ceph.com/docs/next/install/os-recommendations/

 Do you think you can provide an update for dumpling.

 Best Regards

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Supported by Citrix XenServer Yet?

[John Wilkins]

Larry,

If you're talking about how to do that with libvirt and QEMU on
CentOS6.4, you might look at
http://openstack.redhat.com/Using_Ceph_for_Block_Storage_with_RDO. You
just don't need to install and configure OpenStack, obviously. You do
need to get the upstream version of QEMU from the Ceph repository
though.



On Thu, Sep 5, 2013 at 10:03 AM, Liu, Larry larry@disney.com wrote:
 Thanks, Neil!  Anyone has a working doc on how to generate a secret for a
 CentOS6.4 tech preview machine to access a RBD cluster?

 From: Neil Levine neil.lev...@inktank.com
 Date: Thursday, August 29, 2013 5:01 PM
 To: Larry Liu larry@disney.com
 Cc: ceph-users@lists.ceph.com ceph-users@lists.ceph.com
 Subject: Re: [ceph-users] Ceph Supported by Citrix XenServer Yet?

 The XenServer product has a tech preview version available with Ceph RBD
 support:

 http://xenserver.org/discuss-virtualization/virtualization-blog/entry/tech-preview-of-xenserver-libvirt-ceph.html

 The fully-supported, commercial version from Citrix will be available
 sometime in Q4.

 Neil



 On Thu, Aug 29, 2013 at 4:55 PM, Liu, Larry larry@disney.com wrote:

 Hi guys,

 Anyone heard anything if Citrix XenServer supports Ceph yet?  Provision
 CentOS6.4 then on the top of it installing Xen seems a bit too much.

 Thanks!

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-deply preflight hostname check?

[John Wilkins]

Let me follow up on that and get back to you. There has been a
significant amount of work on ceph-deploy since that was written.

On Wed, Sep 4, 2013 at 9:27 PM, Nigel Williams
nigel.d.willi...@gmail.com wrote:
 I notice under HOSTNAME RESOLUTION section the use of 'host -4
 {hostname}' as a required test, however, in all my trial deployments
 so far, none would pass as this command is a direct DNS query, and
 instead I usually just add entries to the host file.

 Two thoughts, is Ceph expecting to only do DNS queries? or instead
 would it be better for the pre-flight to use the getent  hosts
 {hostname} as a test?
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Administering a ceph cluster

[John Wilkins]

This is an error in the docs.  Upstart jobs apply to each node. I've
updated the docs to reflect this understanding. When deployed as a
service with the -a option, ceph would start daemons across nodes.
With upstart, you need to start and stop by invoking upstart on each
node.

On Tue, Aug 27, 2013 at 10:03 AM, Francesc Alted franc...@continuum.io wrote:
 Hi,

 So I have already setup a shiny new Ceph cluster (in one single machine,
 quantal64, adminstered from another machine, precise64). Now, for operating
 the cluster, I am a bit unsure on how to interpret the docs in
 http://ceph.com/docs/next/rados/operations/operating/.  My interpretation is
 that I should start the cluster from the *admin* node, right?  But once I
 have done this in precise64 (via `sudo start ceph-all`), I try to see the
 status of it with the `ceph` command and I am getting this:

 $ ceph
 2013-08-27 16:50:35.946904 7f43d44c6700  1 -- :/0 messenger.start
 2013-08-27 16:50:35.947392 7f43d44c6700 -1 monclient(hunting): ERROR:
 missing keyring, cannot use cephx for authentication
 2013-08-27 16:50:35.947410 7f43d44c6700  0 librados: client.admin
 initialization error (2) No such file or directory
 2013-08-27 16:50:35.947444 7f43d44c6700  1 -- :/1020622 mark_down_all
 2013-08-27 16:50:35.947604 7f43d44c6700  1 -- :/1020622 shutdown complete.
 Error connecting to cluster: ObjectNotFound

 Then, I tried to start the cluster right at 'cluster' machine (quantal64),
 but I am getting the same error in the admin machine.  Here it is the
 contents of my 'my-cluster' directory in the admin machine:

 vagrant@precise64:~/my-cluster$ ls
 ceph.bootstrap-mds.keyring  ceph.bootstrap-osd.keyring
 ceph.client.admin.keyring  ceph.conf  ceph.log  ceph.mon.keyring

 and my ceph.conf contents:

 $ cat ceph.conf
 [global]
 fsid = 64b3090b-a692-4993-98a0-ba3e0bedd7db
 mon initial members = quantal64
 mon host = 192.168.33.11
 auth supported = cephx
 osd journal size = 1024
 filestore xattr use omap = true

 [osd.1]
 host = quantal64

 Am I doing something wrong?

 Thanks,

 --
 Francesc Alted
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Deployments

[John Wilkins]

Actually, I wrote the Quick Start guides so that you could do exactly
what you are trying to do, but mostly from a kick the tires
perspective so that people can learn to use Ceph without imposing
$100k worth of hardware as a requirement. See
http://ceph.com/docs/master/start/quick-ceph-deploy/

I even added a section so that you could do it on one disk--e.g., on
your laptop.  
http://ceph.com/docs/master/start/quick-ceph-deploy/#multiple-osds-on-the-os-disk-demo-only

It says demo only, because you won't get great performance out of a
single node. Monitors, OSDs, and Journals writing to disk and fsync
issues would make performance sub-optimal.

For better performance, you should consider a separate drive for each
Ceph OSD Daemon if you can, and potentially a separate SSD drive
partitioned for journals. If you can separate the OS and monitor
drives from the OSD drives, that's better too.

I wrote it as a two-node quick start, because you cannot kernel mount
the Ceph Filesystem or Ceph Block Devices on the same host as the Ceph
Storage Cluster. It's a kernel issue, not a Ceph issue. However, you
can get around this too. If your machine has enough RAM and CPU, you
can also install virtual machines and kernel mount cephfs and block
devices in the virtual machines with no kernel issues. You don't need
to use VMs at all for librbd. So you can install QEMU/KVM, libvirt and
OpenStack all on the same host too.  It's just not an ideal situation
from performance or high availability perspective.



On Mon, Aug 19, 2013 at 3:12 AM, Schmitt, Christian
c.schm...@briefdomain.de wrote:
 2013/8/19 Wolfgang Hennerbichler wolfgang.hennerbich...@risc-software.at:
 On 08/19/2013 12:01 PM, Schmitt, Christian wrote:
 yes. depends on 'everything', but it's possible (though not recommended)
 to run mon, mds, and osd's on the same host, and even do virtualisation.

 Currently we don't want to virtualise on this machine since the
 machine is really small, as said we focus on small to midsize
 businesses. Most of the time they even need a tower server due to the
 lack of a correct rack. ;/

 whoa :)

 Yep that's awful.

 Our Application, Ceph's object storage and a database?

 what is 'a database'?

 We run Postgresql or MariaDB (without/with Galera depending on the cluster 
 size)

 You wouldn't want to put the data of postgres or mariadb on cephfs. I
 would run the native versions directly on the servers and use
 mysql-multi-master circular replication. I don't know about similar
 features of postgres.

 No i don't want to put a MariaDB Cluster on CephFS we want to put PDFs
 in CephFS or Ceph's Object Storage and hold a key or path in the
 database, also other things like user management will belong to the
 database

 shared nothing is possible with ceph, but in the end this really depends
 on your application.

 hm, when disk fails we already doing some backup on a dell powervault
 rd1000, so i don't think thats a problem and also we would run ceph on
 a Dell PERC Raid Controller with RAID1 enabled on the data disk.

 this is open to discussion, and really depends on your use case.

 Yeah we definitely know that it isn't good to use Ceph on a single
 node, but i think it's easier to design the application that it will
 depends on ceph. it wouldn't be easy to manage to have a single node
 without ceph and more than 1 node with ceph.

 Currently we make an archiving software for small customers and we want
 to move things on the file system on a object storage.

 you mean from the filesystem to an object storage?

 yes, currently everything is on the filesystem and this is really
 horrible, thousands of pdfs just on the filesystem. we can't scale up
 that easily with this setup.

 Got it.

 Currently we run on Microsoft Servers, but we plan to rewrite our
 whole codebase with scaling in mind, from 1 to X Servers. So 1, 3, 5,
 7, 9, ... X²-1 should be possible.

 cool.

 Currently we only
 have customers that needs 1 machine or 3 machines. But everything should
 work as fine on more.

 it would with ceph. probably :)

 That's nice to hear. I was really scared that we don't find a solution
 that can run on 1 system and scale up to even more. We first looked at
 HDFS but this isn't lightweight.

 not only that, HDFS also has a single point of failure.

 And the overhead of Metadata etc.
 just isn't that cool.

 :)

 Yeah that's why I came to Ceph. I think that's probably the way we want to go.
 Really thank you for your help. It's good to know that I have a
 solution for the things that are badly designed on our current
 solution.

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http

Re: [ceph-users] Ceph Deployments

[John Wilkins]

Wolfgang is correct. You do not need VMs at all if you are setting up
Ceph Object Storage. It's just Apache, FastCGI, and the radosgw daemon
interacting with the Ceph Storage Cluster. You can do that on one box
no problem. It's still better to have more drives for performance
though.

On Mon, Aug 19, 2013 at 12:08 PM, Wolfgang Hennerbichler
wolfgang.hennerbich...@risc-software.at wrote:
 What you are trying to do will work, because you will not need any kernel 
 related code for object storage, so a one node setup will work for you.

 --
 Sent from my mobile device

 On 19.08.2013, at 20:29, Schmitt, Christian c.schm...@briefdomain.de 
 wrote:

 That sounds bad for me.
 As said one of the things we consider is a one node setup, for production.
 Not every Customer will afford hardware worth more than ~4000 Euro.
 Small business users don't need need the biggest hardware, but i don't
 think it's a good way to have a version who uses the filesystem and
 one version who use ceph.

 We prefer a Object Storage for our Files. It should work like the
 Object Storage of the App Engine.
 That scales from 1 to X Servers.


 2013/8/19 John Wilkins john.wilk...@inktank.com:
 Actually, I wrote the Quick Start guides so that you could do exactly
 what you are trying to do, but mostly from a kick the tires
 perspective so that people can learn to use Ceph without imposing
 $100k worth of hardware as a requirement. See
 http://ceph.com/docs/master/start/quick-ceph-deploy/

 I even added a section so that you could do it on one disk--e.g., on
 your laptop.  
 http://ceph.com/docs/master/start/quick-ceph-deploy/#multiple-osds-on-the-os-disk-demo-only

 It says demo only, because you won't get great performance out of a
 single node. Monitors, OSDs, and Journals writing to disk and fsync
 issues would make performance sub-optimal.

 For better performance, you should consider a separate drive for each
 Ceph OSD Daemon if you can, and potentially a separate SSD drive
 partitioned for journals. If you can separate the OS and monitor
 drives from the OSD drives, that's better too.

 I wrote it as a two-node quick start, because you cannot kernel mount
 the Ceph Filesystem or Ceph Block Devices on the same host as the Ceph
 Storage Cluster. It's a kernel issue, not a Ceph issue. However, you
 can get around this too. If your machine has enough RAM and CPU, you
 can also install virtual machines and kernel mount cephfs and block
 devices in the virtual machines with no kernel issues. You don't need
 to use VMs at all for librbd. So you can install QEMU/KVM, libvirt and
 OpenStack all on the same host too.  It's just not an ideal situation
 from performance or high availability perspective.



 On Mon, Aug 19, 2013 at 3:12 AM, Schmitt, Christian
 c.schm...@briefdomain.de wrote:
 2013/8/19 Wolfgang Hennerbichler wolfgang.hennerbich...@risc-software.at:
 On 08/19/2013 12:01 PM, Schmitt, Christian wrote:
 yes. depends on 'everything', but it's possible (though not recommended)
 to run mon, mds, and osd's on the same host, and even do virtualisation.

 Currently we don't want to virtualise on this machine since the
 machine is really small, as said we focus on small to midsize
 businesses. Most of the time they even need a tower server due to the
 lack of a correct rack. ;/

 whoa :)

 Yep that's awful.

 Our Application, Ceph's object storage and a database?

 what is 'a database'?

 We run Postgresql or MariaDB (without/with Galera depending on the 
 cluster size)

 You wouldn't want to put the data of postgres or mariadb on cephfs. I
 would run the native versions directly on the servers and use
 mysql-multi-master circular replication. I don't know about similar
 features of postgres.

 No i don't want to put a MariaDB Cluster on CephFS we want to put PDFs
 in CephFS or Ceph's Object Storage and hold a key or path in the
 database, also other things like user management will belong to the
 database

 shared nothing is possible with ceph, but in the end this really depends
 on your application.

 hm, when disk fails we already doing some backup on a dell powervault
 rd1000, so i don't think thats a problem and also we would run ceph on
 a Dell PERC Raid Controller with RAID1 enabled on the data disk.

 this is open to discussion, and really depends on your use case.

 Yeah we definitely know that it isn't good to use Ceph on a single
 node, but i think it's easier to design the application that it will
 depends on ceph. it wouldn't be easy to manage to have a single node
 without ceph and more than 1 node with ceph.

 Currently we make an archiving software for small customers and we want
 to move things on the file system on a object storage.

 you mean from the filesystem to an object storage?

 yes, currently everything is on the filesystem and this is really
 horrible, thousands of pdfs just on the filesystem. we can't scale up
 that easily with this setup.

 Got it.

 Currently we run on Microsoft Servers, but we plan

Re: [ceph-users] FW: Issues with ceph-deploy

[John Wilkins]

/listinfo.cgi/ceph-users-ceph.com





-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Basic questions

[John Wilkins]

(a) This is true when using ceph-deploy for a cluster. It's one Ceph
Monitor for the cluster on one node. You can have many Ceph monitors,
but the typical high availability cluster has 3-5 monitor nodes. With
a manual install, you could conceivably install multiple monitors onto
a single node for the same cluster, but this isn't a best practice
since the node is a failure domain. The monitor is part of the
cluster, not the node. So you can have thousands of nodes running Ceph
daemons that are members of the cluster ceph. A node that has a
monitor for cluster ceph will monitor all Ceph OSD daemons and MDS
daemons across those thousands of nodes. That same node could also
have a monitor for cluster deep-storage or whatever cluster name you
choose.

(b) I'm actually working on a reference architecture for Calxeda that
is asking exactly that question. My personal feeling is that having a
machine/host/chassis optimized for a particular purpose (e.g., running
Ceph OSDs) is the ideal scenario, since you can just add hardware to
the cluster to expand it. You don't need to add monitors or MDSs to
add OSDs. So my personal opinion is that it's an ideal approach. The
upcoming Calxeda offerings provide excellent value in the
cost/performance tradeoff. You get a lot of storage density and good
performance. High performance clusters--e.g., using SSDs for journals,
having more RAM and CPU power--cost more, but you still have some of
the same issues. I still don't have a firm opinion on this, but my gut
tells me that OSDs should be separate from the other daemons--build
OSD hosts with dense storage. The fsync issues with the
kernel--running monitors and OSDs on the same host--generally lead to
performance issues. See
http://ceph.com/docs/master/rados/troubleshooting/troubleshooting-osd/#osds-are-slow-unresponsive
for examples of why you may run into performance issues making
different types of processes co-resident on the same host.  Processes
like monitors shouldn't be co-resident with OSDs. So you don't have
wasted hosts with light weight processes like Ceph monitors, it may be
ideal to place your MDS daemons, Apache/RGW daemons,
OpenStack/CloudStack, and/or VMs on those nodes. You need to consider
the CPU, RAM, disk i/o and network implications of co-resident
applications.

(d) If you have three monitors, Paxos will still work. 2 out of 3
monitors is a majority. A failure of a monitor means it's down, but
not out. If it were out of the cluster, then the cluster would assume
only two monitors, which wouldn't work with Paxos. That's why 3
monitors is the minimum for high availability. 4 works too, because 3
out of 4 is a majority too. Some people like using an odd number of
monitors, since you never have an equal number of monitors that are
up/down; however, this isn't a requirement for Paxos. 3 out of 4 and 3
out of 5 both constitute a majority.





On Fri, Jul 26, 2013 at 11:29 AM, Hariharan Thantry than...@gmail.com wrote:
 Hi John,

 Thanks for the responses.

 For (a), I remember reading somewhere that one can only run a max of 1
 monitor/node, I assume that that implies the single monitor process will be
 responsible for ALL ceph clusters on that node, correct?

 So (b) isn't really a Ceph issue, that's nice to know. Any recommendations
 on the minimum kernel/glibc version and min RAM size requirements where Ceph
 can be run on a single client in native mode? Reason I ask this is in a few
 deployment scenarios (especially non-standard like telco platforms),
 hardware gets added gradually, so its more important to be able to scale the
 cluster out gracefully. I actually see Ceph as an alternative to SAN, using
 JBODs from machines to create a larg(ish) storage cluster. Plus, usually,
 the clients would probably be running on the same hardware as the OSD/MON,
 because space on the chassis is at a premium.

 (d) I was thinking about single node failure scenarios, with 3 nodes,
 wouldn't a failure of 1 node cause PAXOS to not work?



 Thanks,
 Hari





 On Fri, Jul 26, 2013 at 10:00 AM, John Wilkins john.wilk...@inktank.com
 wrote:

 (a) Yes. See
 http://ceph.com/docs/master/rados/configuration/ceph-conf/#running-multiple-clusters
 and
 http://ceph.com/docs/master/rados/deployment/ceph-deploy-new/#naming-a-cluster
 (b) Yes. See
 http://wiki.ceph.com/03FAQs/01General_FAQ#How_Can_I_Give_Ceph_a_Try.3F
  Mounting kernel modules on the same node as Ceph Daemons can cause
 older kernels to deadlock.
 (c) Someone else can probably answer that better than me.
 (d) At least three. Paxos requires a simple majority, so 2 out of 3 is
 sufficient. See
 http://ceph.com/docs/master/rados/configuration/mon-config-ref/#background
 particularly the monitor quorum section.

 On Wed, Jul 24, 2013 at 4:03 PM, Hariharan Thantry than...@gmail.com
 wrote:
  Hi folks,
 
  Some very basic questions.
 
  (a) Can I be running more than 1 ceph cluster on the same node (assume
  that
  I have no more than 1 monitor/node, but storage

Re: [ceph-users] Getting started with RADOS Gateway

[John Wilkins]

 --uid=gwuser1



 { user_id: gwuser1,



   display_name: {Rados Gateway User1},



   email: ,



   suspended: 0,



   max_buckets: 1000,



   auid: 0,



   subusers: [],



   keys: [



 { user: gwuser1,



   access_key: B5CL8KIB8LYH55FDPYPS,



   secret_key: B5\/EEMuTaTkSU5sS1zol2OknBPch04ZZdh86GGTA}],



   swift_keys: [



 { user: gwuser1:swift,



   secret_key: kVcKPg1QFu73emgrP7w6JrFs3tvZ4+gLISW+703K}],



   caps: []}







 # tail -30 /etc/httpd/conf/httpd.conf



 # The first VirtualHost section is used for requests without a known



 # server name.



 #



 #FastCgiExternalServer /var/www/s3gw.fcgi -socket /tmp/radosgw.sock







 VirtualHost *:80



 ServerAdmin chris.howa...@citi.com



 DocumentRoot /var/www



 ServerName e8c3-dl360g7-09.nam.nsroot.net



 #ErrorLog logs/dummy-host.example.com-error_log



 #CustomLog logs/dummy-host.example.com-access_log common



 RewriteEngine On



 RewriteRule ^/([a-zA-Z0-9-_.]*)([/]?.*)

 /s3gw.fcgi?page=$1params=$2%{QUERY_STRING}

 [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]



 IfModule mod_fastcgi.c



 Directory /var/www



 Options +ExecCGI



 AllowOverride All



 SetHandler fastcgi-script



 Order allow,deny



 Allow from all



 AuthBasicAuthoritative Off



 /Directory



 FastCgiExternalServer /var/www/s3gw.fcgi -socket /tmp/radosgw.sock



 /IfModule



 AllowEncodedSlashes On



 ErrorLog /var/log/httpd/error.log



 CustomLog /var/log/httpd/access.log combined



 ServerSignature Off







 /VirtualHost







 # cat /var/www/s3gw.fcgi



 #!/bin/sh



 exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.gateway



 __



 Chris Howarth



 OS Platforms Engineering



 Citi Architecture  Technology Engineering



 (e) chris.howa...@citi.com



 (t) +44 (0) 20 7508 3848



 (f) +44 (0) 20 7508 0964



 (mail-drop) CGC-06-3A









 ___

 ceph-users mailing list

 ceph-users@lists.ceph.com

 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com





 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Python APIs

[John Wilkins]

Actually no.  I'll write up an API doc for you soon.

sudo apt-get install python-ceph

import rados

You can view the code by cloning the git repository.
http://ceph.com/docs/master/install/clone-source/

The source is in src/pybind/rados.py.

See http://ceph.com/docs/master/rbd/librbdpy/

The first lines are what you need to connect.

cluster = rados.Rados(conffile='my_ceph.conf')
cluster.connect()
ioctx = cluster.open_ioctx('mypool')

Also, you will need to modify your ceph.conf file to include a
keyring. Put it under global.

[global]
...
keyring=path/to/keyring/ceph.client.admin.keyring

The rest of the process is similar to the C API.

http://ceph.com/docs/master/rados/api/librados/

Hope this helps. I'll get something up for you soon.

Regards,


John



















On Wed, Jun 12, 2013 at 12:34 AM, Giuseppe \Gippa\ Paternò
gpate...@gpaterno.com wrote:
 Is using s3/swift emulation the only way to access object store with python?


 On 06/11/2013 08:32 PM, John Wilkins wrote:
 Here are the libraries for the Ceph Object Store.

 http://ceph.com/docs/master/radosgw/s3/python/
 http://ceph.com/docs/master/radosgw/swift/python/






--
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-deploy questions

[John Wilkins]

ceph-deploy adds the OSDs to the cluster map. You can add the OSDs to
the ceph.conf manually.

In the ceph.conf file, the settings don't require underscores. If you
modify your configuration at runtime, you need to add the underscores
on the command line.

http://ceph.com/docs/master/rados/configuration/ceph-conf/
http://ceph.com/docs/master/rados/configuration/ceph-conf/#runtime-changes

Underscores and dashes work with the config settings.

On Tue, Jun 11, 2013 at 4:41 PM, Scottix scot...@gmail.com wrote:
 Hi Everyone,
 I am new to ceph but loving every moment of it. I am learning all of this
 now, so maybe this will help with documentation.

 Anyway, I have a few question about ceph-deploy. I was able to setup a
 cluster and be able to get it up and running no problem with ubuntu 12.04.2
 that isn't the problem. The ceph.conf file is a little bit of a mystery for
 me on ceph-deploy. For example when I create a mon or osd on a machine the
 ceph.conf file doesn't change at all. Then if I reboot an osd, I have to
 re-activate it every time. Am I suppose to edit the
 config file for each osd? If I don't edit the file how do I keep track of
 each machine? or set special parameters for some machines? or does it
 matter?

 One last thing is why does it put underscores '_' for spaces when it does
 deploy the ceph.conf? Seems odd since the documentation doesn't show
 underscores, but I guess it doesn't matter since it works.

 Thanks for clarification,
 Scottix

 --
 Follow Me: @Scottix

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Python APIs

[John Wilkins]

Here are the libraries for the Ceph Object Store.

http://ceph.com/docs/master/radosgw/s3/python/
http://ceph.com/docs/master/radosgw/swift/python/

On Tue, Jun 11, 2013 at 2:17 AM, Giuseppe \Gippa\ Paternò
gpate...@gpaterno.com wrote:
 Hi! Sorry for the dumb question, could you point me out to the Python
 APIs reference docs for the object store?
 Do you have example to share for reading files/dirs?
 Thanks,
 Giuseppe
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RDB

[John Wilkins]

Gary,

I've added that instruction to the docs. It should be up shortly. Let
me know if you have other feedback for the docs.

Regards,

John

On Mon, Jun 10, 2013 at 9:13 AM, Gary Bruce garyofscotl...@gmail.com wrote:
 Hi again,

 I don't see anything in http://ceph.com/docs/master/start/ that mentions
 installing ceph-common or a package that would have it as a dependency on
 the admin server. If there's a gap in the documentation, I'd like to help
 address it.

 If I need to install ceph-common on my admin node, how should I go about
 doing it as this is not clear from the documentation. Some possible
 approaches are to run one of these commands from my admin node, cephadmin1:

 *** sudo apt-get install ceph-common
 *** sudo apt-get install ceph
 *** ceph-deploy install --stable cuttlefish cephadmin1// I used
 ceph-deploy install --stable cuttlefish cephserver1 to install ceph on my
 server node from my admin node.

 Any thoughts on the most appropriate way to install ceph-common (and other
 required packages) on cephadmin?

 Thanks
 Gary


 On Sun, Jun 9, 2013 at 10:03 AM, Smart Weblications GmbH
 i...@smart-weblications.de wrote:

 Hi,

 Am 09.06.2013 10:42, schrieb Gary Bruce:
  Hi,
 
  I'm trying to run this from my admin node, have I missed a step?
 
 
  alphaceph@cephadmin1:~/ceph-deploy/my-cluster$ rbd create fooimage
  --size 1024
  --pool barpool -m cephserver1.zion.bt.co.uk
  http://cephserver1.zion.bt.co.uk
  -k /etc/ceph/ceph.client.admin.keyring


 Look:

  The program 'rbd' is currently not installed. To run 'rbd' please ask
  your
  administrator to install the package 'ceph-common'
 

 Maybe you missed installing ceph-common on your host cephadmin1



 --

 Mit freundlichen Grüßen,


 Smart Weblications GmbH
 Martinsberger Str. 1
 D-95119 Naila

 fon.: +49 9282 9638 200
 fax.: +49 9282 9638 205
 24/7: +49 900 144 000 00 - 0,99 EUR/Min*
 http://www.smart-weblications.de

 --
 Sitz der Gesellschaft: Naila
 Geschäftsführer: Florian Wiessner
 HRB-Nr.: HRB 3840 Amtsgericht Hof
 *aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz



 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-deploy

[John Wilkins]

Actually, as I said, I unmounted them first, zapped the disk, then
used OSD create. For you, that might look like:

sudo umount /dev/sda3
ceph-deploy disk zap  ceph0:sda3 ceph1:sda3 ceph2:sda3
ceph-deploy osd create ceph0:sda3 ceph1:sda3 ceph2:sda3

I was referring to the entire disk in my deployment, but I wasn't
using partitions on the same disk. So ceph-deploy created the data and
journal partitions for me. If you are running multiple OSDs on the
same disk (not recommended, except for evaluation), you'd want to use
the following procedure:


On Sat, Jun 1, 2013 at 7:57 AM, Dewan Shamsul Alam
dewan.sham...@gmail.com wrote:
 Hi John,

 I have a feeling that I am missing something. Previously when I succeeded
 with bobtail with mkcephfs, I mounted the /dev/sdb1 partitions. There is
 nothing mentioned in the blog about it though.

 Say I have 3 nodes ceph201 ceph202 and ceph 203. Each has a /dev/sdb1
 partition formatted as xfs. Do I need to mount them in a particular
 directory prior running the command or ceph-deploy would take care of it?


 On Thu, May 30, 2013 at 8:17 PM, John Wilkins john.wilk...@inktank.com
 wrote:

 Dewan,

 I encountered this too. I just did umount and reran the command and it
 worked for me. I probably need to add a troubleshooting section for
 ceph-deploy.

 On Fri, May 24, 2013 at 4:00 PM, John Wilkins john.wilk...@inktank.com
 wrote:
  ceph-deploy does have an ability to push the client keyrings. I
  haven't encountered this as a problem. However, I have created a
  monitor and not seen it return a keyring. In other words, it failed
  but didn't give me a warning message. So I just re-executed creating
  the monitor. The directory from where you execute ceph-deploy mon
  create should have a ceph.client.admin.keyring too. If it doesn't,
  you might have had a problem creating the monitor. I don't believe you
  have to push the ceph.client.admin.keyring to all the nodes. So it
  shouldn't be barking back unless you failed to create the monitor, or
  if gatherkeys failed.
 
  On Thu, May 23, 2013 at 9:09 PM, Dewan Shamsul Alam
  dewan.sham...@gmail.com wrote:
  I just found that
 
  #ceph-deploy gatherkeys ceph0 ceph1 ceph2
 
  works only if I have bobtail. cuttlefish can't find ceph.client.admin.
  keyring
 
  and then when I try this on bobtail, it says,
 
  root@cephdeploy:~/12.04# ceph-deploy osd create ceph0:/dev/sda3
  ceph1:/dev/sda3 ceph2:/dev/sda3
  ceph-disk: Error: Device is mounted: /dev/sda3
  Traceback (most recent call last):
File /usr/bin/ceph-deploy, line 22, in module
  main()
File /usr/lib/pymodules/python2.7/ceph_deploy/cli.py, line 112, in
  main
  return args.func(args)
File /usr/lib/pymodules/python2.7/ceph_deploy/osd.py, line 293, in
  osd
  prepare(args, cfg, activate_prepared_disk=True)
File /usr/lib/pymodules/python2.7/ceph_deploy/osd.py, line 177, in
  prepare
  dmcrypt_dir=args.dmcrypt_key_dir,
File /usr/lib/python2.7/dist-packages/pushy/protocol/proxy.py, line
  255,
  in lambda
  (conn.operator(type_, self, args, kwargs))
File /usr/lib/python2.7/dist-packages/pushy/protocol/connection.py,
  line
  66, in operator
  return self.send_request(type_, (object, args, kwargs))
File
  /usr/lib/python2.7/dist-packages/pushy/protocol/baseconnection.py,
  line 323, in send_request
  return self.__handle(m)
File
  /usr/lib/python2.7/dist-packages/pushy/protocol/baseconnection.py,
  line 639, in __handle
  raise e
  pushy.protocol.proxy.ExceptionProxy: Command '['ceph-disk-prepare',
  '--',
  '/dev/sda3']' returned non-zero exit status 1
  root@cephdeploy:~/12.04#
 
 
 
 
  On Thu, May 23, 2013 at 10:49 PM, Dewan Shamsul Alam
  dewan.sham...@gmail.com wrote:
 
  Hi,
 
  I tried ceph-deploy all day. Found that it has a python-setuptools as
  dependency. I knew about python-pushy. But is there any other
  dependency
  that I'm missing?
 
  The problem I'm getting are as follows:
 
  #ceph-deploy gatherkeys ceph0 ceph1 ceph2
  returns the following error,
  Unable to find /etc/ceph/ceph.client.admin.keyring on ['ceph0',
  'ceph1',
  'ceph2']
 
  Once I got passed this, I don't know why it works sometimes. I have
  been
  following the exact steps as mentioned in the blog.
 
  Then when I try to do
 
  ceph-deploy osd create ceph0:/dev/sda3 ceph1:/dev/sda3 ceph2:/dev/sda3
 
  It gets stuck.
 
  I'm using Ubuntu 13.04 for ceph-deploy and 12.04 for ceph nodes. I
  just
  need to get the cuttlefish working and willing to change the OS if it
  is
  required. Please help. :)
 
  Best Regards,
  Dewan Shamsul Alam
 
 
 
  ___
  ceph-users mailing list
  ceph-users@lists.ceph.com
  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
 
 
  --
  John Wilkins
  Senior Technical Writer
  Intank
  john.wilk...@inktank.com
  (415) 425-9599
  http://inktank.com



 --
 John Wilkins
 Senior Technical Writer
 Intank
 john.wilk...@inktank.com
 (415) 425-9599
 http

Re: [ceph-users] ceph-deploy

[John Wilkins]

Sorry...hit send inadvertantly...

http://ceph.com/docs/master/start/quick-ceph-deploy/#multiple-osds-on-the-os-disk-demo-only

On Mon, Jun 3, 2013 at 1:00 PM, John Wilkins john.wilk...@inktank.com wrote:
 Actually, as I said, I unmounted them first, zapped the disk, then
 used OSD create. For you, that might look like:

 sudo umount /dev/sda3
 ceph-deploy disk zap  ceph0:sda3 ceph1:sda3 ceph2:sda3
 ceph-deploy osd create ceph0:sda3 ceph1:sda3 ceph2:sda3

 I was referring to the entire disk in my deployment, but I wasn't
 using partitions on the same disk. So ceph-deploy created the data and
 journal partitions for me. If you are running multiple OSDs on the
 same disk (not recommended, except for evaluation), you'd want to use
 the following procedure:


 On Sat, Jun 1, 2013 at 7:57 AM, Dewan Shamsul Alam
 dewan.sham...@gmail.com wrote:
 Hi John,

 I have a feeling that I am missing something. Previously when I succeeded
 with bobtail with mkcephfs, I mounted the /dev/sdb1 partitions. There is
 nothing mentioned in the blog about it though.

 Say I have 3 nodes ceph201 ceph202 and ceph 203. Each has a /dev/sdb1
 partition formatted as xfs. Do I need to mount them in a particular
 directory prior running the command or ceph-deploy would take care of it?


 On Thu, May 30, 2013 at 8:17 PM, John Wilkins john.wilk...@inktank.com
 wrote:

 Dewan,

 I encountered this too. I just did umount and reran the command and it
 worked for me. I probably need to add a troubleshooting section for
 ceph-deploy.

 On Fri, May 24, 2013 at 4:00 PM, John Wilkins john.wilk...@inktank.com
 wrote:
  ceph-deploy does have an ability to push the client keyrings. I
  haven't encountered this as a problem. However, I have created a
  monitor and not seen it return a keyring. In other words, it failed
  but didn't give me a warning message. So I just re-executed creating
  the monitor. The directory from where you execute ceph-deploy mon
  create should have a ceph.client.admin.keyring too. If it doesn't,
  you might have had a problem creating the monitor. I don't believe you
  have to push the ceph.client.admin.keyring to all the nodes. So it
  shouldn't be barking back unless you failed to create the monitor, or
  if gatherkeys failed.
 
  On Thu, May 23, 2013 at 9:09 PM, Dewan Shamsul Alam
  dewan.sham...@gmail.com wrote:
  I just found that
 
  #ceph-deploy gatherkeys ceph0 ceph1 ceph2
 
  works only if I have bobtail. cuttlefish can't find ceph.client.admin.
  keyring
 
  and then when I try this on bobtail, it says,
 
  root@cephdeploy:~/12.04# ceph-deploy osd create ceph0:/dev/sda3
  ceph1:/dev/sda3 ceph2:/dev/sda3
  ceph-disk: Error: Device is mounted: /dev/sda3
  Traceback (most recent call last):
File /usr/bin/ceph-deploy, line 22, in module
  main()
File /usr/lib/pymodules/python2.7/ceph_deploy/cli.py, line 112, in
  main
  return args.func(args)
File /usr/lib/pymodules/python2.7/ceph_deploy/osd.py, line 293, in
  osd
  prepare(args, cfg, activate_prepared_disk=True)
File /usr/lib/pymodules/python2.7/ceph_deploy/osd.py, line 177, in
  prepare
  dmcrypt_dir=args.dmcrypt_key_dir,
File /usr/lib/python2.7/dist-packages/pushy/protocol/proxy.py, line
  255,
  in lambda
  (conn.operator(type_, self, args, kwargs))
File /usr/lib/python2.7/dist-packages/pushy/protocol/connection.py,
  line
  66, in operator
  return self.send_request(type_, (object, args, kwargs))
File
  /usr/lib/python2.7/dist-packages/pushy/protocol/baseconnection.py,
  line 323, in send_request
  return self.__handle(m)
File
  /usr/lib/python2.7/dist-packages/pushy/protocol/baseconnection.py,
  line 639, in __handle
  raise e
  pushy.protocol.proxy.ExceptionProxy: Command '['ceph-disk-prepare',
  '--',
  '/dev/sda3']' returned non-zero exit status 1
  root@cephdeploy:~/12.04#
 
 
 
 
  On Thu, May 23, 2013 at 10:49 PM, Dewan Shamsul Alam
  dewan.sham...@gmail.com wrote:
 
  Hi,
 
  I tried ceph-deploy all day. Found that it has a python-setuptools as
  dependency. I knew about python-pushy. But is there any other
  dependency
  that I'm missing?
 
  The problem I'm getting are as follows:
 
  #ceph-deploy gatherkeys ceph0 ceph1 ceph2
  returns the following error,
  Unable to find /etc/ceph/ceph.client.admin.keyring on ['ceph0',
  'ceph1',
  'ceph2']
 
  Once I got passed this, I don't know why it works sometimes. I have
  been
  following the exact steps as mentioned in the blog.
 
  Then when I try to do
 
  ceph-deploy osd create ceph0:/dev/sda3 ceph1:/dev/sda3 ceph2:/dev/sda3
 
  It gets stuck.
 
  I'm using Ubuntu 13.04 for ceph-deploy and 12.04 for ceph nodes. I
  just
  need to get the cuttlefish working and willing to change the OS if it
  is
  required. Please help. :)
 
  Best Regards,
  Dewan Shamsul Alam
 
 
 
  ___
  ceph-users mailing list
  ceph-users@lists.ceph.com
  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RADOS Gateway Configuration

[John Wilkins]

Cool. I did the same thing with Cuttlefish at one point. I scrubbed my
install and started the whole thing--even the storage cluster--from
scratch after doing an update. There might have been a bug in the mix
that got fixed, because I was scratching my head too and after I did
the whole re-install everything worked perfectly.

On Fri, May 31, 2013 at 9:30 AM, Daniel Curran danielcurra...@gmail.com wrote:
 Hey,

 Just so you know I got sick of messing with it and decided to try bobtail.
 It worked with the exact steps we've been referring to. I'm sure it's
 something silly that changed making my stuff not work.

 Thanks,
 Daniel


 On Thu, May 30, 2013 at 11:23 AM, Daniel Curran danielcurra...@gmail.com
 wrote:

 Here's what it looks like at the moment. The permissions were 600 but I
 changed them and recreated the user/subuser/key to no avail.

 $ ls -lah /etc/ceph/
 total 28K
 drwxr-xr-x   2 root root 4.0K May 29 15:58 .
 drwxr-xr-x 131 root root  12K May 28 15:21 ..
 -rw-r--r--   1 root root   64 May 28 15:19 ceph.client.admin.keyring
 -rw-r--r--   1 root root  362 May 29 15:54 ceph.conf
 -rw-r--r--   1 root root  119 May 28 15:19 keyring.radosgw.gateway



 On Thu, May 30, 2013 at 9:26 AM, John Wilkins john.wilk...@inktank.com
 wrote:

 Do you have your admin keyring in the /etc/ceph directory of your
 radosgw host?  That sounds like step 1 here:
 http://ceph.com/docs/master/start/quick-rgw/#generate-a-keyring-and-key

 I think I encountered an issue there myself, and did a sudo chmod 644
 on the keyring.

 On Wed, May 29, 2013 at 1:17 PM, Daniel Curran danielcurra...@gmail.com
 wrote:
  Unfortunately it seems like I messed up yesterday. I didn't have the
  client.radosgw.gateway section in my ceph.conf. I don't get the apache
  errors now but I still don't have access since the secret_key is still
  not
  being created or at least not showing up. I can try to auth but it just
  says
  'Auth GET failed: http://192.168.1.100:80/auth/ 403 Forbidden' with
  everything I send it.
 
  This is what I have at the moment in the files you requested.
  ceph.conf:
  --
  --
  [global]
  fsid = 1ec4438a-3f59-4cfd-86b8-a89607401d81
  mon_initial_members = ceph0
  mon_host = 192.168.1.100
  auth_supported = cephx
  osd_journal_size = 1024
  filestore_xattr_use_omap = true
 
  [client.radosgw.gateway]
  host = ceph0
  keyring = /etc/ceph/keyring.radosgw.gateway
  rgw socket path = /tmp/radosgw.sock
  log file = /var/log/ceph/radosgw.log
  rgw dns name = ceph0
 
  
 
  rgw.conf:
 
  
  FastCgiExternalServer /var/www/s3gw.fcgi -socket /tmp/radosgw.sock
 
 
  VirtualHost *:80
  ServerName ceph0
  ServerAdmin admin@localhost
  DocumentRoot /var/www
 
 
  RewriteEngine On
  RewriteRule ^/([a-zA-Z0-9-_.]*)([/]?.*)
  /s3gw.fcgi?page=$1params=$2%{QUERY_STRING}
  [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
 
  IfModule mod_fastcgi.c
  Directory /var/www
  Options +ExecCGI
  AllowOverride All
  SetHandler fastcgi-script
  Order allow,deny
  Allow from all
  AuthBasicAuthoritative Off
  /Directory
  /IfModule
 
  AllowEncodedSlashes On
  ErrorLog /var/log/apache2/error.log
  CustomLog /var/log/apache2/access.log combined
  ServerSignature Off
 
  /VirtualHost
 
  
 
  s3gw.fcgi
 
  
 
  #!/bin/sh
  exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.gateway
 
  
 
  Here's what the /var/log/ceph/radosgw.log says when it denies me:
  2013-05-29 16:00:29.118234 7f5f60cf6700  2 req
  11:0.93:swift-auth:GET
  /auth/::getting op
  2013-05-29 16:00:29.118237 7f5f60cf6700  2 req
  11:0.96:swift-auth:GET
  /auth/:swift_auth_get:authorizing
  2013-05-29 16:00:29.118239 7f5f60cf6700  2 req
  11:0.98:swift-auth:GET
  /auth/:swift_auth_get:reading permissions
  2013-05-29 16:00:29.118243 7f5f60cf6700  2 req
  11:0.000102:swift-auth:GET
  /auth/:swift_auth_get:reading the cors attr
  2013-05-29 16:00:29.118246 7f5f60cf6700 10 Going to read cors from
  attrs
  2013-05-29 16:00:29.118248 7f5f60cf6700  2 req
  11:0.000107:swift-auth:GET
  /auth/:swift_auth_get:verifying op permissions
  2013-05-29 16:00:29.118250 7f5f60cf6700  2 req
  11:0.000109:swift-auth:GET
  /auth/:swift_auth_get:verifying op params
  2013-05-29 16:00

Re: [ceph-users] RADOS Gateway Configuration

[John Wilkins]

Ah. I was using the S3 interface. Yes, that's what I did and
Cuttlefish worked for me. If you're working on Bobtail, I'd stick with
that for evaluation. We'll have a new update to Cuttlefish shortly.

On Fri, May 31, 2013 at 1:02 PM, Daniel Curran danielcurra...@gmail.com wrote:
 Do you mean that Cuttlefish worked for you after starting from scratch? I've
 been working from fresh ubuntu instances each time so I've redone everything
 multiple times. After a bunch of failures the only thing that worked for me
 was avoiding Cuttlefish altogether and using Bobtail instead.
 I think there may still be a bug in Cuttlefish which doesn't set the value
 of secret_key in the swift_keys map that's there or something. That's
 the only thing I can think of at least.

 Daniel


 On Fri, May 31, 2013 at 12:17 PM, John Wilkins john.wilk...@inktank.com
 wrote:

 Cool. I did the same thing with Cuttlefish at one point. I scrubbed my
 install and started the whole thing--even the storage cluster--from
 scratch after doing an update. There might have been a bug in the mix
 that got fixed, because I was scratching my head too and after I did
 the whole re-install everything worked perfectly.

 On Fri, May 31, 2013 at 9:30 AM, Daniel Curran danielcurra...@gmail.com
 wrote:
  Hey,
 
  Just so you know I got sick of messing with it and decided to try
  bobtail.
  It worked with the exact steps we've been referring to. I'm sure it's
  something silly that changed making my stuff not work.
 
  Thanks,
  Daniel
 
 
  On Thu, May 30, 2013 at 11:23 AM, Daniel Curran
  danielcurra...@gmail.com
  wrote:
 
  Here's what it looks like at the moment. The permissions were 600 but I
  changed them and recreated the user/subuser/key to no avail.
 
  $ ls -lah /etc/ceph/
  total 28K
  drwxr-xr-x   2 root root 4.0K May 29 15:58 .
  drwxr-xr-x 131 root root  12K May 28 15:21 ..
  -rw-r--r--   1 root root   64 May 28 15:19 ceph.client.admin.keyring
  -rw-r--r--   1 root root  362 May 29 15:54 ceph.conf
  -rw-r--r--   1 root root  119 May 28 15:19 keyring.radosgw.gateway
 
 
 
  On Thu, May 30, 2013 at 9:26 AM, John Wilkins
  john.wilk...@inktank.com
  wrote:
 
  Do you have your admin keyring in the /etc/ceph directory of your
  radosgw host?  That sounds like step 1 here:
 
  http://ceph.com/docs/master/start/quick-rgw/#generate-a-keyring-and-key
 
  I think I encountered an issue there myself, and did a sudo chmod 644
  on the keyring.
 
  On Wed, May 29, 2013 at 1:17 PM, Daniel Curran
  danielcurra...@gmail.com
  wrote:
   Unfortunately it seems like I messed up yesterday. I didn't have the
   client.radosgw.gateway section in my ceph.conf. I don't get the
   apache
   errors now but I still don't have access since the secret_key is
   still
   not
   being created or at least not showing up. I can try to auth but it
   just
   says
   'Auth GET failed: http://192.168.1.100:80/auth/ 403 Forbidden' with
   everything I send it.
  
   This is what I have at the moment in the files you requested.
   ceph.conf:
   --
   --
   [global]
   fsid = 1ec4438a-3f59-4cfd-86b8-a89607401d81
   mon_initial_members = ceph0
   mon_host = 192.168.1.100
   auth_supported = cephx
   osd_journal_size = 1024
   filestore_xattr_use_omap = true
  
   [client.radosgw.gateway]
   host = ceph0
   keyring = /etc/ceph/keyring.radosgw.gateway
   rgw socket path = /tmp/radosgw.sock
   log file = /var/log/ceph/radosgw.log
   rgw dns name = ceph0
  
  
   
  
   rgw.conf:
  
  
   
   FastCgiExternalServer /var/www/s3gw.fcgi -socket /tmp/radosgw.sock
  
  
   VirtualHost *:80
   ServerName ceph0
   ServerAdmin admin@localhost
   DocumentRoot /var/www
  
  
   RewriteEngine On
   RewriteRule ^/([a-zA-Z0-9-_.]*)([/]?.*)
   /s3gw.fcgi?page=$1params=$2%{QUERY_STRING}
   [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
  
   IfModule mod_fastcgi.c
   Directory /var/www
   Options +ExecCGI
   AllowOverride All
   SetHandler fastcgi-script
   Order allow,deny
   Allow from all
   AuthBasicAuthoritative Off
   /Directory
   /IfModule
  
   AllowEncodedSlashes On
   ErrorLog /var/log/apache2/error.log
   CustomLog /var/log/apache2/access.log combined
   ServerSignature Off
  
   /VirtualHost
  
  
   
  
   s3gw.fcgi
  
  
   
  
   #!/bin/sh
   exec /usr/bin/radosgw -c /etc/ceph

Re: [ceph-users] ceph-deploy

[John Wilkins]

Dewan,

I encountered this too. I just did umount and reran the command and it
worked for me. I probably need to add a troubleshooting section for
ceph-deploy.

On Fri, May 24, 2013 at 4:00 PM, John Wilkins john.wilk...@inktank.com wrote:
 ceph-deploy does have an ability to push the client keyrings. I
 haven't encountered this as a problem. However, I have created a
 monitor and not seen it return a keyring. In other words, it failed
 but didn't give me a warning message. So I just re-executed creating
 the monitor. The directory from where you execute ceph-deploy mon
 create should have a ceph.client.admin.keyring too. If it doesn't,
 you might have had a problem creating the monitor. I don't believe you
 have to push the ceph.client.admin.keyring to all the nodes. So it
 shouldn't be barking back unless you failed to create the monitor, or
 if gatherkeys failed.

 On Thu, May 23, 2013 at 9:09 PM, Dewan Shamsul Alam
 dewan.sham...@gmail.com wrote:
 I just found that

 #ceph-deploy gatherkeys ceph0 ceph1 ceph2

 works only if I have bobtail. cuttlefish can't find ceph.client.admin.
 keyring

 and then when I try this on bobtail, it says,

 root@cephdeploy:~/12.04# ceph-deploy osd create ceph0:/dev/sda3
 ceph1:/dev/sda3 ceph2:/dev/sda3
 ceph-disk: Error: Device is mounted: /dev/sda3
 Traceback (most recent call last):
   File /usr/bin/ceph-deploy, line 22, in module
 main()
   File /usr/lib/pymodules/python2.7/ceph_deploy/cli.py, line 112, in main
 return args.func(args)
   File /usr/lib/pymodules/python2.7/ceph_deploy/osd.py, line 293, in osd
 prepare(args, cfg, activate_prepared_disk=True)
   File /usr/lib/pymodules/python2.7/ceph_deploy/osd.py, line 177, in
 prepare
 dmcrypt_dir=args.dmcrypt_key_dir,
   File /usr/lib/python2.7/dist-packages/pushy/protocol/proxy.py, line 255,
 in lambda
 (conn.operator(type_, self, args, kwargs))
   File /usr/lib/python2.7/dist-packages/pushy/protocol/connection.py, line
 66, in operator
 return self.send_request(type_, (object, args, kwargs))
   File /usr/lib/python2.7/dist-packages/pushy/protocol/baseconnection.py,
 line 323, in send_request
 return self.__handle(m)
   File /usr/lib/python2.7/dist-packages/pushy/protocol/baseconnection.py,
 line 639, in __handle
 raise e
 pushy.protocol.proxy.ExceptionProxy: Command '['ceph-disk-prepare', '--',
 '/dev/sda3']' returned non-zero exit status 1
 root@cephdeploy:~/12.04#




 On Thu, May 23, 2013 at 10:49 PM, Dewan Shamsul Alam
 dewan.sham...@gmail.com wrote:

 Hi,

 I tried ceph-deploy all day. Found that it has a python-setuptools as
 dependency. I knew about python-pushy. But is there any other dependency
 that I'm missing?

 The problem I'm getting are as follows:

 #ceph-deploy gatherkeys ceph0 ceph1 ceph2
 returns the following error,
 Unable to find /etc/ceph/ceph.client.admin.keyring on ['ceph0', 'ceph1',
 'ceph2']

 Once I got passed this, I don't know why it works sometimes. I have been
 following the exact steps as mentioned in the blog.

 Then when I try to do

 ceph-deploy osd create ceph0:/dev/sda3 ceph1:/dev/sda3 ceph2:/dev/sda3

 It gets stuck.

 I'm using Ubuntu 13.04 for ceph-deploy and 12.04 for ceph nodes. I just
 need to get the cuttlefish working and willing to change the OS if it is
 required. Please help. :)

 Best Regards,
 Dewan Shamsul Alam



 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




 --
 John Wilkins
 Senior Technical Writer
 Intank
 john.wilk...@inktank.com
 (415) 425-9599
 http://inktank.com



-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RADOS Gateway Configuration

[John Wilkins]

-05-29 16:00:29.118380 7f5f60cf6700  2 req 11:0.000238:swift-auth:GET
 /auth/:swift_auth_get:http status=403
 2013-05-29 16:00:29.118504 7f5f60cf6700  1 == req done req=0x14d3b30
 http_status=403 ==


 Which looks reasonable since I don't have the proper key to send along to
 it. Was there a specific log file you wanted to see?

 Thanks,
 Daniel


 On Tue, May 28, 2013 at 6:50 PM, John Wilkins john.wilk...@inktank.com
 wrote:

 Can you paste your ceph.conf file, rgw.conf file, s3gw.fcgi file and
 your log file?

 I'm trying to work out a reliable procedure for the S3 subdomain, and
 ran into a a few issues.

 On Tue, May 28, 2013 at 4:45 PM, Daniel Curran danielcurra...@gmail.com
 wrote:
  Hmmit looks like maybe my issue is bigger than just the swift
  secret_key
  not showing up. I started using the other key that's provided under
  keys
  in the user map just to see if it works but I get 500's back. In my
  apache
  error log I get
 
  [Tue May 28 15:19:59 2013] [notice] Apache/2.2.22 (Ubuntu)
  mod_fastcgi/mod_fastcgi-SNAP-0910052141 configured -- resuming normal
  operations
  [Tue May 28 15:24:04 2013] [error] [client 192.168.1.100] (2)No such
  file or
  directory: FastCGI: failed to connect to server /var/www/s3gw.fcgi:
  connect() failed
  [Tue May 28 15:24:04 2013] [error] [client 192.168.1.100] FastCGI:
  incomplete headers (0 bytes) received from server /var/www/s3gw.fcgi
 
  The errors pop up when I try to hit that endpoint. It's strange because
  it's
  owned by www-data which apache is using and has r/x permissions for
  everyone. I'm thinking maybe there's something wrong in my rewrite rule
  or
  with the s3gw.fcgi itself.
 
  rewrite rules:
  RewriteEngine On
  RewriteRule ^/([a-zA-Z0-9-_.]*)([/]?.*)
  /s3gw.fcgi?page=$1params=$2%{QUERY_STRING}
  [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
 
  contents of /var/www/s3gw.fcgi:
  #!/bin/sh
  exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.gateway
 
 
 
 
 
 
 
 
 
  On Tue, May 28, 2013 at 11:39 AM, Daniel Curran
  danielcurra...@gmail.com
  wrote:
 
  Hey,
 
  I'll check see if I can get it going right now. Thanks for taking the
  time
  to help out
 
  Daniel
 
 
  On Sat, May 25, 2013 at 7:01 PM, John Wilkins
  john.wilk...@inktank.com
  wrote:
 
  Daniel,
 
  I finished up the procedure this morning and put it up on the web. It
  seems to be working fine for me.
 
  On Fri, May 24, 2013 at 3:35 PM, John Wilkins
  john.wilk...@inktank.com
  wrote:
   Daniel,
  
   I'm working on it. It did need a bit of a rewrite for ceph-deploy,
   even though the steps are conceptually the same. In the step here:
  
  
  
   http://ceph.com/docs/master/start/quick-rgw/#modify-the-ceph-configuration-file
  
   1. The rewrite--not published yet, as I'm working out a few more
   kinks--is to use the ceph-deploy version of the ceph conf file on
   the
   admin node. So you make the modification to the configuration file
   there; then, use ceph-deploy config push to push it to the server
   host.
  
   2. Create a directory is on the server node, as before.
  
   3. The rewrite rule there is outside of the VirtualHost tags, and
   there are two instances of VirtualHost. Remove the first end, and
   second beginning of the /VirtualHostVirtualHost *:80, but leave
   the rewrite rule. That was a bug. Sorry.
  
   4. Then, create the keyring here:
  
   http://ceph.com/docs/master/start/quick-rgw/#generate-a-keyring-and-key
It wouldn't have worked before, because you didn't have the right
   keyring in /etc/ceph. The third step in that procedure says
   ceph.keyring.  That's no longer correct. By convention with
   ceph-deploy, the key ring is ceph.client.admin.keyring.
  
   5. http://ceph.com/docs/master/start/quick-rgw/#enable-ssl  Do this
   before creating the users if you want to use SSL.
  
   6. Restart ceph, apache, and radosgw as the root user.  Then,
   proceed
   to creating a user.
  
   I'll add some examples when I get all the quirks ironed out so you
   can
   just cut and paste.
  
   Regards,
  
  
   John
  
  
   On Fri, May 24, 2013 at 2:22 PM, Daniel Curran
   danielcurra...@gmail.com wrote:
  
  
   -- Forwarded message --
   From: Daniel Curran danielcurra...@gmail.com
   Date: Fri, May 24, 2013 at 4:21 PM
   Subject: Re: [ceph-users] RADOS Gateway Configuration
   To: John Wilkins john.wilk...@inktank.com
  
  
   Hey,
  
   I just finished the 5-minute quick start for rados that was given
   earlier
   but I get the same issue.
  
  
   { user_id: johndoe,
 display_name: John Doe,
 email: j...@example.com,
 suspended: 0,
 max_buckets: 1000,
 auid: 0,
 subusers: [
   { id: johndoe:swift,
 permissions: full-control}],
 keys: [
   { user: johndoe,
 access_key: ZM3S10OX6SPBDPGZC19J,
 secret_key:
   AhZlWa1cLkff\/v0NLnMb7HGIAcPu3lMgM8CTKxey}],
  
 swift_keys: [
   { user: johndoe:swift

Re: [ceph-users] RADOS Gateway Configuration

[John Wilkins]

Can you paste your ceph.conf file, rgw.conf file, s3gw.fcgi file and
your log file?

I'm trying to work out a reliable procedure for the S3 subdomain, and
ran into a a few issues.

On Tue, May 28, 2013 at 4:45 PM, Daniel Curran danielcurra...@gmail.com wrote:
 Hmmit looks like maybe my issue is bigger than just the swift secret_key
 not showing up. I started using the other key that's provided under keys
 in the user map just to see if it works but I get 500's back. In my apache
 error log I get

 [Tue May 28 15:19:59 2013] [notice] Apache/2.2.22 (Ubuntu)
 mod_fastcgi/mod_fastcgi-SNAP-0910052141 configured -- resuming normal
 operations
 [Tue May 28 15:24:04 2013] [error] [client 192.168.1.100] (2)No such file or
 directory: FastCGI: failed to connect to server /var/www/s3gw.fcgi:
 connect() failed
 [Tue May 28 15:24:04 2013] [error] [client 192.168.1.100] FastCGI:
 incomplete headers (0 bytes) received from server /var/www/s3gw.fcgi

 The errors pop up when I try to hit that endpoint. It's strange because it's
 owned by www-data which apache is using and has r/x permissions for
 everyone. I'm thinking maybe there's something wrong in my rewrite rule or
 with the s3gw.fcgi itself.

 rewrite rules:
 RewriteEngine On
 RewriteRule ^/([a-zA-Z0-9-_.]*)([/]?.*)
 /s3gw.fcgi?page=$1params=$2%{QUERY_STRING}
 [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]

 contents of /var/www/s3gw.fcgi:
 #!/bin/sh
 exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.gateway









 On Tue, May 28, 2013 at 11:39 AM, Daniel Curran danielcurra...@gmail.com
 wrote:

 Hey,

 I'll check see if I can get it going right now. Thanks for taking the time
 to help out

 Daniel


 On Sat, May 25, 2013 at 7:01 PM, John Wilkins john.wilk...@inktank.com
 wrote:

 Daniel,

 I finished up the procedure this morning and put it up on the web. It
 seems to be working fine for me.

 On Fri, May 24, 2013 at 3:35 PM, John Wilkins john.wilk...@inktank.com
 wrote:
  Daniel,
 
  I'm working on it. It did need a bit of a rewrite for ceph-deploy,
  even though the steps are conceptually the same. In the step here:
 
 
  http://ceph.com/docs/master/start/quick-rgw/#modify-the-ceph-configuration-file
 
  1. The rewrite--not published yet, as I'm working out a few more
  kinks--is to use the ceph-deploy version of the ceph conf file on the
  admin node. So you make the modification to the configuration file
  there; then, use ceph-deploy config push to push it to the server
  host.
 
  2. Create a directory is on the server node, as before.
 
  3. The rewrite rule there is outside of the VirtualHost tags, and
  there are two instances of VirtualHost. Remove the first end, and
  second beginning of the /VirtualHostVirtualHost *:80, but leave
  the rewrite rule. That was a bug. Sorry.
 
  4. Then, create the keyring here:
  http://ceph.com/docs/master/start/quick-rgw/#generate-a-keyring-and-key
   It wouldn't have worked before, because you didn't have the right
  keyring in /etc/ceph. The third step in that procedure says
  ceph.keyring.  That's no longer correct. By convention with
  ceph-deploy, the key ring is ceph.client.admin.keyring.
 
  5. http://ceph.com/docs/master/start/quick-rgw/#enable-ssl  Do this
  before creating the users if you want to use SSL.
 
  6. Restart ceph, apache, and radosgw as the root user.  Then, proceed
  to creating a user.
 
  I'll add some examples when I get all the quirks ironed out so you can
  just cut and paste.
 
  Regards,
 
 
  John
 
 
  On Fri, May 24, 2013 at 2:22 PM, Daniel Curran
  danielcurra...@gmail.com wrote:
 
 
  -- Forwarded message --
  From: Daniel Curran danielcurra...@gmail.com
  Date: Fri, May 24, 2013 at 4:21 PM
  Subject: Re: [ceph-users] RADOS Gateway Configuration
  To: John Wilkins john.wilk...@inktank.com
 
 
  Hey,
 
  I just finished the 5-minute quick start for rados that was given
  earlier
  but I get the same issue.
 
 
  { user_id: johndoe,
display_name: John Doe,
email: j...@example.com,
suspended: 0,
max_buckets: 1000,
auid: 0,
subusers: [
  { id: johndoe:swift,
permissions: full-control}],
keys: [
  { user: johndoe,
access_key: ZM3S10OX6SPBDPGZC19J,
secret_key: AhZlWa1cLkff\/v0NLnMb7HGIAcPu3lMgM8CTKxey}],
 
swift_keys: [
  { user: johndoe:swift,
secret_key: }],
caps: []}
 
  with no secret_key under swift_keys for me to pass along.
  Regenerating the
  keys doesn't do anything different. I've also tried using the
  secret_key
  under keys but that doesn't work either. Any thoughts?
 
 
  Thanks your help,
  Daniel
 
 
 
 
  On Thu, May 23, 2013 at 1:28 PM, John Wilkins
  john.wilk...@inktank.com
  wrote:
 
  It puts it in the same directory where you executed ceph-deploy.
 
  On Thu, May 23, 2013 at 10:57 AM, Daniel Curran
  danielcurra...@gmail.com wrote:
   Hey John,
  
   Thanks for the reply. I'll check out that other doc you have there.
   Just

Re: [ceph-users] RADOS Gateway Configuration

[John Wilkins]

Daniel,

I finished up the procedure this morning and put it up on the web. It
seems to be working fine for me.

On Fri, May 24, 2013 at 3:35 PM, John Wilkins john.wilk...@inktank.com wrote:
 Daniel,

 I'm working on it. It did need a bit of a rewrite for ceph-deploy,
 even though the steps are conceptually the same. In the step here:

 http://ceph.com/docs/master/start/quick-rgw/#modify-the-ceph-configuration-file

 1. The rewrite--not published yet, as I'm working out a few more
 kinks--is to use the ceph-deploy version of the ceph conf file on the
 admin node. So you make the modification to the configuration file
 there; then, use ceph-deploy config push to push it to the server
 host.

 2. Create a directory is on the server node, as before.

 3. The rewrite rule there is outside of the VirtualHost tags, and
 there are two instances of VirtualHost. Remove the first end, and
 second beginning of the /VirtualHostVirtualHost *:80, but leave
 the rewrite rule. That was a bug. Sorry.

 4. Then, create the keyring here:
 http://ceph.com/docs/master/start/quick-rgw/#generate-a-keyring-and-key
  It wouldn't have worked before, because you didn't have the right
 keyring in /etc/ceph. The third step in that procedure says
 ceph.keyring.  That's no longer correct. By convention with
 ceph-deploy, the key ring is ceph.client.admin.keyring.

 5. http://ceph.com/docs/master/start/quick-rgw/#enable-ssl  Do this
 before creating the users if you want to use SSL.

 6. Restart ceph, apache, and radosgw as the root user.  Then, proceed
 to creating a user.

 I'll add some examples when I get all the quirks ironed out so you can
 just cut and paste.

 Regards,


 John


 On Fri, May 24, 2013 at 2:22 PM, Daniel Curran danielcurra...@gmail.com 
 wrote:


 -- Forwarded message --
 From: Daniel Curran danielcurra...@gmail.com
 Date: Fri, May 24, 2013 at 4:21 PM
 Subject: Re: [ceph-users] RADOS Gateway Configuration
 To: John Wilkins john.wilk...@inktank.com


 Hey,

 I just finished the 5-minute quick start for rados that was given earlier
 but I get the same issue.


 { user_id: johndoe,
   display_name: John Doe,
   email: j...@example.com,
   suspended: 0,
   max_buckets: 1000,
   auid: 0,
   subusers: [
 { id: johndoe:swift,
   permissions: full-control}],
   keys: [
 { user: johndoe,
   access_key: ZM3S10OX6SPBDPGZC19J,
   secret_key: AhZlWa1cLkff\/v0NLnMb7HGIAcPu3lMgM8CTKxey}],

   swift_keys: [
 { user: johndoe:swift,
   secret_key: }],
   caps: []}

 with no secret_key under swift_keys for me to pass along. Regenerating the
 keys doesn't do anything different. I've also tried using the secret_key
 under keys but that doesn't work either. Any thoughts?


 Thanks your help,
 Daniel




 On Thu, May 23, 2013 at 1:28 PM, John Wilkins john.wilk...@inktank.com
 wrote:

 It puts it in the same directory where you executed ceph-deploy.

 On Thu, May 23, 2013 at 10:57 AM, Daniel Curran
 danielcurra...@gmail.com wrote:
  Hey John,
 
  Thanks for the reply. I'll check out that other doc you have there. Just
  for
  future reference do you know where ceph-deploy puts the ceph keyring?
 
  Daniel
 
 
  On Wed, May 22, 2013 at 7:19 PM, John Wilkins john.wilk...@inktank.com
  wrote:
 
  Daniel,
 
  It looks like I need to update that portion of the docs too, as it
  links back to the 5-minute quick start. Once you are up and running
  with HEALTH OK on either the 5-minute Quick Start or Quick Ceph
  Deploy, your storage cluster is running fine. The remaining issues
  would likely be with authentication, chmod on the files, or with the
  RGW setup. There's a quick start for RGW, which I had verified here:
  http://ceph.com/docs/master/start/quick-rgw/. Someone else had a
  problem with the Rewrite rule on that example reported here:
  http://tracker.ceph.com/issues/4608. It's likely I need to run through
  with specific Ceph and Apache versions. There are also a few
  additional tips in the configuration section.
  http://ceph.com/docs/master/radosgw/config/
 
  There is an issue in some cases where keys have forward or backslash
  characters, and you may need to regenerate the keys.
 
 
 
  On Wed, May 22, 2013 at 4:42 PM, Daniel Curran
  danielcurra...@gmail.com
  wrote:
  
   Hello,
  
   I just started using ceph recently and was trying to get the RADOS
   Gateway
   working in order to use the Swift compatible API. I followed the
   install
   instructions found here (http://ceph.com/docs/master
   /start/quick-ceph-deploy/) and got to a point where ceph health
   give
   me
   HEALTH_OK. This is all well and good but near the end of the rados gw
   setup
   (found here http://ceph.com/docs/master/radosgw/manual-install/) I
   need
   to
   execute the following line:
  
   sudo ceph -k /etc/ceph/ceph.keyring auth add client.radosgw.gateway
   -i
   /etc/ceph/keyring.radosgw.gateway
  
   Unfortunately, I don't believe ceph-deploy places the keyring

Re: [ceph-users] RADOS Gateway Configuration

[John Wilkins]

It puts it in the same directory where you executed ceph-deploy.

On Thu, May 23, 2013 at 10:57 AM, Daniel Curran
danielcurra...@gmail.com wrote:
 Hey John,

 Thanks for the reply. I'll check out that other doc you have there. Just for
 future reference do you know where ceph-deploy puts the ceph keyring?

 Daniel


 On Wed, May 22, 2013 at 7:19 PM, John Wilkins john.wilk...@inktank.com
 wrote:

 Daniel,

 It looks like I need to update that portion of the docs too, as it
 links back to the 5-minute quick start. Once you are up and running
 with HEALTH OK on either the 5-minute Quick Start or Quick Ceph
 Deploy, your storage cluster is running fine. The remaining issues
 would likely be with authentication, chmod on the files, or with the
 RGW setup. There's a quick start for RGW, which I had verified here:
 http://ceph.com/docs/master/start/quick-rgw/. Someone else had a
 problem with the Rewrite rule on that example reported here:
 http://tracker.ceph.com/issues/4608. It's likely I need to run through
 with specific Ceph and Apache versions. There are also a few
 additional tips in the configuration section.
 http://ceph.com/docs/master/radosgw/config/

 There is an issue in some cases where keys have forward or backslash
 characters, and you may need to regenerate the keys.



 On Wed, May 22, 2013 at 4:42 PM, Daniel Curran danielcurra...@gmail.com
 wrote:
 
  Hello,
 
  I just started using ceph recently and was trying to get the RADOS
  Gateway
  working in order to use the Swift compatible API. I followed the install
  instructions found here (http://ceph.com/docs/master
  /start/quick-ceph-deploy/) and got to a point where ceph health give
  me
  HEALTH_OK. This is all well and good but near the end of the rados gw
  setup
  (found here http://ceph.com/docs/master/radosgw/manual-install/) I need
  to
  execute the following line:
 
  sudo ceph -k /etc/ceph/ceph.keyring auth add client.radosgw.gateway -i
  /etc/ceph/keyring.radosgw.gateway
 
  Unfortunately, I don't believe ceph-deploy places the keyring at
  /etc/ceph/ceph.keyring. I tried to use the one from
  /var/lib/ceph/bootstrap-osd/ceph.keyring but it was unable to
  authenticate
  as client.admin. Is there another location that the keyring needs to be
  copied from or am I doing something totally wrong?
 
  I didn't want to be held back so I restarted and did the manual install
  from
  the 5-minute quick start where I was able to find the ring. I had more
  issues almost immediately. I have to execute the following steps to
  create
  some users for swift:
 
  radosgw-admin user create --uid=johndoe --display-name=John Doe
  --email=j...@example.com
  sudo radosgw-admin subuser create --uid=johndoe --subuser=johndoe:swift
  --access=full
 
  sudo radosgw-admin key create --subuser=johndoe:swift --key-type=swift
 
  The first two gave me output I was expecting but the very last line had
  some
  weirdness that essentially made swift unusable. The expected output is
  something along these lines:
 
  { user_id: johndoe,
rados_uid: 0,
display_name: John Doe,
email: j...@example.com,
suspended: 0,
subusers: [
   { id: johndoe:swift,
 permissions: full-control}],
keys: [
  { user: johndoe,
access_key: QFAMEDSJP5DEKJO0DDXY,
secret_key: iaSFLDVvDdQt6lkNzHyW4fPLZugBAI1g17LO0+87}],
swift_keys: [
  { user: johndoe:swift,
secret_key: E9T2rUZNu2gxUjcwUBO8n\/Ev4KX6\/GprEuH4qhu1}]}
 
  Where that last secret key is what we hand the swift CLI as seen here:
 
  swift -V 1.0 -A http://radosgw.example.com/auth -U johndoe:swift -K
  E9T2rUZNu2gxUjcwUBO8n\/Ev4KX6\/GprEuH4qhu1 post test
 
  However, my output came out like this:
 
  { user_id: johndoe,
display_name: John Doe,
email: j...@example.com,
suspended: 0,
max_buckets: 1000,
auid: 0,
   subusers: [
   { id: johndoe:swift,
 permissions: full-control}],
keys: [
  { user: johndoe,
access_key: SUEXWVL3WB2Z64CRAG97,
secret_key: C\/jHFJ3wdPv4iJ+aq4JeZ52LEC3OdnhsYEnVkhBP}],
swift_keys: [
  { user: johndoe:swift,
secret_key: }],
caps: []}
 
 
  Giving me no swift key to use. I don't believe the key is supposed to be
  blank because I tried that and received auth errors (to the best of my
  ability). I can't tell if this is my fault since I'm new nor am I able
  to
  find a way around it. It looks like there are definitely changes between
  the
  version used in the doc and mine so maybe it's all working as it should
  but
  the secret_key for swift lives somewhere else. If anyone knows anything
  I'd
  appreciate it a lot.
 
  Thank you,
  Daniel
 
 
 
  ___
  ceph-users mailing list
  ceph-users@lists.ceph.com
  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 



 --
 John Wilkins
 Senior Technical Writer
 Intank
 john.wilk...@inktank.com
 (415) 425-9599
 http://inktank.com





-- 
John Wilkins
Senior Technical

Re: [ceph-users] PG down incomplete

[John Wilkins]

If you can follow the documentation here:
http://ceph.com/docs/master/rados/operations/monitoring-osd-pg/  and
http://ceph.com/docs/master/rados/troubleshooting/  to provide some
additional information, we may be better able to help you.

For example, ceph osd tree would help us understand the status of
your cluster a bit better.

On Thu, May 16, 2013 at 10:32 PM, Olivier Bonvalet ceph.l...@daevel.fr wrote:
 Le mercredi 15 mai 2013 à 00:15 +0200, Olivier Bonvalet a écrit :
 Hi,

 I have some PG in state down and/or incomplete on my cluster, because I
 loose 2 OSD and a pool was having only 2 replicas. So of course that
 data is lost.

 My problem now is that I can't retreive a HEALTH_OK status : if I try
 to remove, read or overwrite the corresponding RBD images, near all OSD
 hang (well... they don't do anything and requests stay in a growing
 queue, until the production will be done).

 So, what can I do to remove that corrupts images ?

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


 Up. Nobody can help me on that problem ?

 Thanks,

 Olivier

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Mount error 5 while mounting cephfs

[John Wilkins]

Have you tried restarting your MDS server?
http://ceph.com/docs/master/rados/operations/operating/#operating-a-cluster

On Fri, May 17, 2013 at 12:16 AM, Sridhar Mahadevan
msridha...@gmail.com wrote:
 Hi,

 I have deployed the ceph object store using ceph-deploy.
 I tried to mount cephfs and I got struck with this error.

 sudo mount.ceph 192.168.35.82:/ /mnt/mycephfs -o
 name=admin,secret=AQDa5JJRqLxuOxAA77VljIjaAGWR6mGdL12NUQ==

 mount error 5 = Input/output error

 The output of the command

 # ceph -s

health HEALTH_WARN 64 pgs degraded; mds cluster is degraded; mds
 blade2-qq is laggy
monmap e1: 1 mons at {blade2-qq=192.168.35.82:6789/0}, election epoch 1,
 quorum 0 blade2-qq
osdmap e56: 4 osds: 4 up, 4 in
pgmap v834: 192 pgs: 128 active+clean, 64 active+clean+degraded; 0 bytes
 data, 50871 MB used, 437 GB / 513 GB avail
mdsmap e6457: 1/1/1 up {0=blade2-qq=up:replay(laggy or crashed)}

 As it says the MDS has crashed. I dont see ceph-mds running in the MDS_Node.
 I executed ceph-deploy mds create mds_node  and this starts the ceph-mds
 daemon in the mds_node, but I see that the ceph-mds daemon crashes after
 sometime.

 Kindly help me on this issue.

 --
 --sridhar

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Mount error 5 while mounting cephfs

[John Wilkins]

Are you running the MDS in a VM?

On Fri, May 17, 2013 at 12:40 AM, Sridhar Mahadevan
msridha...@gmail.com wrote:
 Hi,
 I did try to restart the MDS server. The logs show the following error

 [187846.234448] init: ceph-mds (ceph/blade2-qq) main process (15077) killed
 by ABRT signal
 [187846.234493] init: ceph-mds (ceph/blade2-qq) main process ended,
 respawning
 [187846.687929] init: ceph-mds (ceph/blade2-qq) main process (15099) killed
 by ABRT signal
 [187846.687977] init: ceph-mds (ceph/blade2-qq) respawning too fast, stopped


 Thanks and Regards


 On Fri, May 17, 2013 at 3:33 PM, John Wilkins john.wilk...@inktank.com
 wrote:

 Have you tried restarting your MDS server?

 http://ceph.com/docs/master/rados/operations/operating/#operating-a-cluster

 On Fri, May 17, 2013 at 12:16 AM, Sridhar Mahadevan
 msridha...@gmail.com wrote:
  Hi,
 
  I have deployed the ceph object store using ceph-deploy.
  I tried to mount cephfs and I got struck with this error.
 
  sudo mount.ceph 192.168.35.82:/ /mnt/mycephfs -o
  name=admin,secret=AQDa5JJRqLxuOxAA77VljIjaAGWR6mGdL12NUQ==
 
  mount error 5 = Input/output error
 
  The output of the command
 
  # ceph -s
 
 health HEALTH_WARN 64 pgs degraded; mds cluster is degraded; mds
  blade2-qq is laggy
 monmap e1: 1 mons at {blade2-qq=192.168.35.82:6789/0}, election epoch
  1,
  quorum 0 blade2-qq
 osdmap e56: 4 osds: 4 up, 4 in
 pgmap v834: 192 pgs: 128 active+clean, 64 active+clean+degraded; 0
  bytes
  data, 50871 MB used, 437 GB / 513 GB avail
 mdsmap e6457: 1/1/1 up {0=blade2-qq=up:replay(laggy or crashed)}
 
  As it says the MDS has crashed. I dont see ceph-mds running in the
  MDS_Node.
  I executed ceph-deploy mds create mds_node  and this starts the
  ceph-mds
  daemon in the mds_node, but I see that the ceph-mds daemon crashes after
  sometime.
 
  Kindly help me on this issue.
 
  --
  --sridhar
 
  ___
  ceph-users mailing list
  ceph-users@lists.ceph.com
  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 



 --
 John Wilkins
 Senior Technical Writer
 Intank
 john.wilk...@inktank.com
 (415) 425-9599
 http://inktank.com




 --
 --sridhar



-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] PG down incomplete

[John Wilkins]

 acting [32,13]
 pg 8.2ff is stuck inactive since forever, current state incomplete, last
 acting [6,19]
 pg 8.5e2 is stuck inactive since forever, current state incomplete, last
 acting [0,19]
 pg 8.528 is stuck inactive since forever, current state incomplete, last
 acting [31,19]
 pg 8.20f is stuck inactive since forever, current state incomplete, last
 acting [31,19]
 pg 8.372 is stuck inactive since forever, current state incomplete, last
 acting [19,24]
 pg 4.5c is stuck unclean since forever, current state incomplete, last
 acting [19,30]
 pg 8.71d is stuck unclean since forever, current state incomplete, last
 acting [24,19]
 pg 8.3fa is stuck unclean since forever, current state incomplete, last
 acting [19,31]
 pg 8.3e0 is stuck unclean since forever, current state incomplete, last
 acting [31,19]
 pg 8.56c is stuck unclean since forever, current state incomplete, last
 acting [19,28]
 pg 8.19f is stuck unclean since forever, current state incomplete, last
 acting [31,19]
 pg 8.792 is stuck unclean since forever, current state incomplete, last
 acting [19,28]
 pg 4.0 is stuck unclean since forever, current state incomplete, last
 acting [28,19]
 pg 8.78a is stuck unclean since forever, current state incomplete, last
 acting [31,19]
 pg 8.23e is stuck unclean since forever, current state incomplete, last
 acting [32,13]
 pg 8.2ff is stuck unclean since forever, current state incomplete, last
 acting [6,19]
 pg 8.5e2 is stuck unclean since forever, current state incomplete, last
 acting [0,19]
 pg 8.528 is stuck unclean since forever, current state incomplete, last
 acting [31,19]
 pg 8.20f is stuck unclean since forever, current state incomplete, last
 acting [31,19]
 pg 8.372 is stuck unclean since forever, current state incomplete, last
 acting [19,24]
 pg 8.792 is incomplete, acting [19,28]
 pg 8.78a is incomplete, acting [31,19]
 pg 8.71d is incomplete, acting [24,19]
 pg 8.5e2 is incomplete, acting [0,19]
 pg 8.56c is incomplete, acting [19,28]
 pg 8.528 is incomplete, acting [31,19]
 pg 8.3fa is incomplete, acting [19,31]
 pg 8.3e0 is incomplete, acting [31,19]
 pg 8.372 is incomplete, acting [19,24]
 pg 8.2ff is incomplete, acting [6,19]
 pg 8.23e is incomplete, acting [32,13]
 pg 8.20f is incomplete, acting [31,19]
 pg 8.19f is incomplete, acting [31,19]
 pg 3.7c is active+clean+inconsistent, acting [24,13,39]
 pg 3.6b is active+clean+inconsistent, acting [28,23,5]
 pg 4.5c is incomplete, acting [19,30]
 pg 3.d is active+clean+inconsistent, acting [29,4,11]
 pg 4.0 is incomplete, acting [28,19]
 pg 3.1 is active+clean+inconsistent, acting [28,19,5]
 osd.10 is near full at 85%
 19 scrub errors
 noout flag(s) set
 mon.d (rank 4) addr 10.0.0.6:6789/0 is down (out of quorum)


 Pools 4 and 8 have only 2 replica, and pool 3 have 3 replica but
 inconsistent data.

 Thanks in advance.

 Le vendredi 17 mai 2013 à 00:14 -0700, John Wilkins a écrit :
 If you can follow the documentation here:
 http://ceph.com/docs/master/rados/operations/monitoring-osd-pg/  and
 http://ceph.com/docs/master/rados/troubleshooting/  to provide some
 additional information, we may be better able to help you.

 For example, ceph osd tree would help us understand the status of
 your cluster a bit better.

 On Thu, May 16, 2013 at 10:32 PM, Olivier Bonvalet ceph.l...@daevel.fr 
 wrote:
  Le mercredi 15 mai 2013 à 00:15 +0200, Olivier Bonvalet a écrit :
  Hi,
 
  I have some PG in state down and/or incomplete on my cluster, because I
  loose 2 OSD and a pool was having only 2 replicas. So of course that
  data is lost.
 
  My problem now is that I can't retreive a HEALTH_OK status : if I try
  to remove, read or overwrite the corresponding RBD images, near all OSD
  hang (well... they don't do anything and requests stay in a growing
  queue, until the production will be done).
 
  So, what can I do to remove that corrupts images ?
 
  ___
  ceph-users mailing list
  ceph-users@lists.ceph.com
  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
  Up. Nobody can help me on that problem ?
 
  Thanks,
 
  Olivier
 
  ___
  ceph-users mailing list
  ceph-users@lists.ceph.com
  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



 --
 John Wilkins
 Senior Technical Writer
 Intank
 john.wilk...@inktank.com
 (415) 425-9599
 http://inktank.com






-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] PG down incomplete

[John Wilkins]

Another thing... since your osd.10 is near full, your cluster may be
fairly close to capacity for the purposes of rebalancing.  Have a look
at:

http://ceph.com/docs/master/rados/configuration/mon-config-ref/#storage-capacity
http://ceph.com/docs/master/rados/troubleshooting/troubleshooting-osd/#no-free-drive-space

Maybe we can get some others to look at this.  It's not clear to me
why the other OSD crashes after you take osd.25 out. It could be
capacity, but that shouldn't make it crash. Have you tried adding more
OSDs to increase capacity?



On Fri, May 17, 2013 at 11:27 AM, John Wilkins john.wilk...@inktank.com wrote:
 It looks like you have the noout flag set:

 noout flag(s) set; 1 mons down, quorum 0,1,2,3 a,b,c,e
monmap e7: 5 mons at
 {a=10.0.0.1:6789/0,b=10.0.0.2:6789/0,c=10.0.0.5:6789/0,d=10.0.0.6:6789/0,e=10.0.0.3:6789/0},
 election epoch 2584, quorum 0,1,2,3 a,b,c,e
osdmap e82502: 50 osds: 48 up, 48 in

 http://ceph.com/docs/master/rados/troubleshooting/troubleshooting-osd/#stopping-w-out-rebalancing

 If you have down OSDs that don't get marked out, that would certainly
 cause problems. Have you tried restarting the failed OSDs?

 What do the logs look like for osd.15 and osd.25?

 On Fri, May 17, 2013 at 1:31 AM, Olivier Bonvalet ceph.l...@daevel.fr wrote:
 Hi,

 thanks for your answer. In fact I have several different problems, which
 I tried to solve separatly :

 1) I loose 2 OSD, and some pools have only 2 replicas. So some data was
 lost.
 2) One monitor refuse the Cuttlefish upgrade, so I only have 4 of 5
 monitors running.
 3) I have 4 old inconsistent PG that I can't repair.


 So the status :

health HEALTH_ERR 15 pgs incomplete; 4 pgs inconsistent; 15 pgs stuck
 inactive; 15 pgs stuck unclean; 1 near full osd(s); 19 scrub errors;
 noout flag(s) set; 1 mons down, quorum 0,1,2,3 a,b,c,e
monmap e7: 5 mons at
 {a=10.0.0.1:6789/0,b=10.0.0.2:6789/0,c=10.0.0.5:6789/0,d=10.0.0.6:6789/0,e=10.0.0.3:6789/0},
  election epoch 2584, quorum 0,1,2,3 a,b,c,e
osdmap e82502: 50 osds: 48 up, 48 in
 pgmap v12807617: 7824 pgs: 7803 active+clean, 1 active+clean
 +scrubbing, 15 incomplete, 4 active+clean+inconsistent, 1 active+clean
 +scrubbing+deep; 5676 GB data, 18948 GB used, 18315 GB / 37263 GB avail;
 137KB/s rd, 1852KB/s wr, 199op/s
mdsmap e1: 0/0/1 up



 The tree :

 # idweight  type name   up/down reweight
 -8  14.26   root SSDroot
 -27 8   datacenter SSDrbx2
 -26 8   room SSDs25
 -25 8   net SSD188-165-12
 -24 8   rack SSD25B09
 -23 8   host lyll
 46  2   osd.46  up   
1
 47  2   osd.47  up   
1
 48  2   osd.48  up   
1
 49  2   osd.49  up   
1
 -10 4.26datacenter SSDrbx3
 -12 2   room SSDs43
 -13 2   net SSD178-33-122
 -16 2   rack SSD43S01
 -17 2   host kaino
 42  1   osd.42  up   
1
 43  1   osd.43  up   
1
 -22 2.26room SSDs45
 -21 2.26net SSD5-135-138
 -20 2.26rack SSD45F01
 -19 2.26host taman
 44  1.13osd.44  up   
1
 45  1.13osd.45  up   
1
 -9  2   datacenter SSDrbx4
 -11 2   room SSDs52
 -14 2   net SSD176-31-226
 -15 2   rack SSD52B09
 -18 2   host dragan
 40  1   osd.40  up   
1
 41  1   osd.41  up   
1
 -1  33.43   root SASroot
 -10015.9datacenter SASrbx1
 -90 15.9room SASs15
 -72 15.9net SAS188-165-15
 -40 8   rack SAS15B01
 -3  8   host brontes
 0   1   osd.0   up   
1
 1   1   osd.1   up   
1
 2   1   osd.2   up   
1
 3   1   osd.3   up   
1

Re: [ceph-users] Regd: Ceph-deploy

[John Wilkins]

This is usually due to a connectivity issue:
http://ceph.com/docs/master/start/quick-start-preflight/#ensure-connectivity
 Make sure ceph-deploy can access the node where you are trying to
deploy the monitor; then, repeat the ceph-deploy mon create step
again. Then, repeat the ceph-deploy gatherkeys step again.

On Mon, May 13, 2013 at 11:45 PM, Sridhar Mahadevan
msridha...@gmail.com wrote:
 Hi,
 I am trying to setup ceph and I am using ceph-deploy. I am following the
 steps in object store quick guide. When I execute ceph-deploy gatherkeys it
 throws up the following error.

 Unable to find /etc/ceph/ceph.client.admin.keyring
 Unable to find /var/lib/ceph/bootstrap-osd/ceph.keyring
 Unable to find /var/lib/ceph/bootstrap-msd/ceph.keyring

 Kindly help

 Thanks and Regards

 --
 --sridhar

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] e release

[John Wilkins]

I like enteroctopus, but Enope is cool and shorter...

http://en.wikipedia.org/wiki/Sparkling_Enope_Squid


On Fri, May 10, 2013 at 11:31 AM, Sage Weil s...@inktank.com wrote:

 We need a cephalopod name that starts with 'e', and trolling through
 taxonomies seems like a great thing to crowdsource.  :)  So far I've found
 a few latin names, but the main problem is that I can't find a single
 large list of species with the common names listed.  Wikipedia's taxonomy
 seems the best so far, but it's still a lot of browsing required as
 cephalopoda is a huge class.

 The only common name I've found is elbow (elbow squid), but elbow is not a
 very fun name.

 Suggestions welcome!

 elbow (elbow squid)

 https://www.google.com/search?q=enteroctopussource=lnmstbm=ischsa=Xei=pzuNUd37McnjigLfu4D4Dwved=0CAoQ_AUoAQbiw=1916bih=1082#tbm=ischsa=1q=elbow+squidoq=elbow+squidgs_l=img.3..0j0i24.80753.82074.2.82218.11.8.0.3.3.0.72.416.8.8.0...0.0...1c.1.12.img.U2rs4lakA-Abav=on.2,or.r_cp.r_qf.bvm=bv.46340616,d.cGEfp=aa2ea750bee51b45biw=1916bih=1082
  http://en.wikipedia.org/wiki/Bigfin_squid

 http://news.nationalgeographic.com/news/2008/11/081124-giant-squid-magnapinna.html

 enteroctopus (giant octopus)
  http://en.wikipedia.org/wiki/Giant_octopus
  http://eol.org/pages/61628/overview

 https://www.google.com/search?q=enteroctopussource=lnmstbm=ischsa=Xei=pzuNUd37McnjigLfu4D4Dwved=0CAoQ_AUoAQbiw=1916bih=1082

 elegent or elegans (sepia elegans, elegent cuttlefish)
  http://en.wikipedia.org/wiki/File:Sepia_elegans.jpg
  http://en.wikipedia.org/wiki/Sepia_(genus) (see sepia elegans, elegant
 cuttlefish)
   it's another cuttlefish, though, so, meh.

 eledone
  http://eol.org/pages/51263/overview

 https://www.google.com/search?q=enteroctopussource=lnmstbm=ischsa=Xei=pzuNUd37McnjigLfu4D4Dwved=0CAoQ_AUoAQbiw=1916bih=1082#tbm=ischsa=1q=eledoneoq=eledonegs_l=img.3..0l3j0i24.15244.15821.0.15909.7.6.0.0.0.0.145.442.5j1.6.0...0.0...1c.1.12.img.VazRyuNNsiQbav=on.2,or.r_cp.r_qf.bvm=bv.46340616,d.cGEfp=aa2ea750bee51b45biw=1916bih=1082

 euaxoctopus
  http://eol.org/pages/49675/overview

 exannulatus (octopus exannulatus)
  http://eol.org/pages/491114/overview

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] e release

[John Wilkins]

As long as we have a picture. Enteroctopus is giant, which implies
large scale and is what we're about.  I just like Enope, because they
are bio-luminescent.
http://en.wikipedia.org/wiki/Sparkling_Enope_Squid  The pictures are
kind of cool too.

On Fri, May 10, 2013 at 11:47 AM, Yehuda Sadeh yeh...@inktank.com wrote:
 On Fri, May 10, 2013 at 11:31 AM, Sage Weil s...@inktank.com wrote:
 We need a cephalopod name that starts with 'e', and trolling through
 taxonomies seems like a great thing to crowdsource.  :)  So far I've found
 a few latin names, but the main problem is that I can't find a single
 large list of species with the common names listed.  Wikipedia's taxonomy
 seems the best so far, but it's still a lot of browsing required as
 cephalopoda is a huge class.

 The only common name I've found is elbow (elbow squid), but elbow is not a
 very fun name.

 Suggestions welcome!

 When we voted on the theme, I've expanded the cephalopods category to
 also include generic marine creatures for this specific reason. We
 can always choose some non-cephalopod creature if the options don't
 feel right (e.g., Eel).

 Yehuda


 elbow (elbow squid)
  
 https://www.google.com/search?q=enteroctopussource=lnmstbm=ischsa=Xei=pzuNUd37McnjigLfu4D4Dwved=0CAoQ_AUoAQbiw=1916bih=1082#tbm=ischsa=1q=elbow+squidoq=elbow+squidgs_l=img.3..0j0i24.80753.82074.2.82218.11.8.0.3.3.0.72.416.8.8.0...0.0...1c.1.12.img.U2rs4lakA-Abav=on.2,or.r_cp.r_qf.bvm=bv.46340616,d.cGEfp=aa2ea750bee51b45biw=1916bih=1082
  http://en.wikipedia.org/wiki/Bigfin_squid
  
 http://news.nationalgeographic.com/news/2008/11/081124-giant-squid-magnapinna.html

 enteroctopus (giant octopus)
  http://en.wikipedia.org/wiki/Giant_octopus
  http://eol.org/pages/61628/overview
  
 https://www.google.com/search?q=enteroctopussource=lnmstbm=ischsa=Xei=pzuNUd37McnjigLfu4D4Dwved=0CAoQ_AUoAQbiw=1916bih=1082

 elegent or elegans (sepia elegans, elegent cuttlefish)
  http://en.wikipedia.org/wiki/File:Sepia_elegans.jpg
  http://en.wikipedia.org/wiki/Sepia_(genus) (see sepia elegans, elegant 
 cuttlefish)
   it's another cuttlefish, though, so, meh.

 eledone
  http://eol.org/pages/51263/overview
  
 https://www.google.com/search?q=enteroctopussource=lnmstbm=ischsa=Xei=pzuNUd37McnjigLfu4D4Dwved=0CAoQ_AUoAQbiw=1916bih=1082#tbm=ischsa=1q=eledoneoq=eledonegs_l=img.3..0l3j0i24.15244.15821.0.15909.7.6.0.0.0.0.145.442.5j1.6.0...0.0...1c.1.12.img.VazRyuNNsiQbav=on.2,or.r_cp.r_qf.bvm=bv.46340616,d.cGEfp=aa2ea750bee51b45biw=1916bih=1082

 euaxoctopus
  http://eol.org/pages/49675/overview

 exannulatus (octopus exannulatus)
  http://eol.org/pages/491114/overview

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 --
 To unsubscribe from this list: send the line unsubscribe ceph-devel in
 the body of a message to majord...@vger.kernel.org
 More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] HEALTH_WARN after upgrade to cuttlefish

[John Wilkins]

James,

The output says,  monmap e1: 3 mons at {4=
192.168.200.197:6789/0,7=192.168.200.190:6789/0,8=192.168.200.191:6789/0},
election epoch 1104, quorum 0,1,2 4,7,8

It looks like you have six OSDs (0,1,2,4,7,8) with only 3 OSDs running. The
cluster needs a majority. So you'd need 4 of 6 monitors running.


On Wed, May 8, 2013 at 4:32 AM, James Harper
james.har...@bendigoit.com.auwrote:

  On 05/08/2013 08:44 AM, David Zafman wrote:
  
   According to osdmap e504: 4 osds: 2 up, 2 in you have 2 of 4 osds
 that are
  down and out.  That may be the issue.
 
  Also, running 'ceph health detail' will give you specifics on what is
  causing the HEALTH_WARN.
 

 # ceph health detail
 HEALTH_WARN
 mon.4 addr 192.168.200.197:6789/0 has 26% avail disk space -- low disk
 space!

 I guess that's the problem.

 Thanks

 James
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-deploy issues

[John Wilkins]

 DEBUG Distro CentOS codename
 Final, will use sysvinit

 2013-05-06 12:14:15,139 ceph_deploy.gatherkeys DEBUG Checking es-sv01 for
 /etc/ceph/ceph.client.admin.keyring

 2013-05-06 12:14:15,682 ceph_deploy.gatherkeys WARNING Unable to find
 /etc/ceph/ceph.client.admin.keyring on ['es-sv01']

 2013-05-06 12:14:15,683 ceph_deploy.gatherkeys DEBUG Have ceph.mon.keyring
 

 2013-05-06 12:14:15,683 ceph_deploy.gatherkeys DEBUG Checking es-sv01 for
 /var/lib/ceph/bootstrap-osd/ceph.keyring

 2013-05-06 12:14:16,233 ceph_deploy.gatherkeys WARNING Unable to find
 /var/lib/ceph/bootstrap-osd/ceph.keyring on ['es-sv01']

 2013-05-06 12:14:16,233 ceph_deploy.gatherkeys DEBUG Checking es-sv01 for
 /var/lib/ceph/bootstrap-mds/ceph.keyring

 2013-05-06 12:14:16,783 ceph_deploy.gatherkeys WARNING Unable to find
 /var/lib/ceph/bootstrap-mds/ceph.keyring on ['es-sv01']

 ** **

 ** **

 Thanks,

 Jim

 ** **

 ** **

 ** **

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 1-Node cluster with no replication

[John Wilkins]

Guido,

My apologies. I seem to have omitted the PG troubleshooting section from
the index. It has been addressed. See
http://ceph.com/docs/master/rados/troubleshooting/troubleshooting-pg/

Ceph OSDs peer and check on each other. So running a cluster with only one
OSD is not recommended. Operationally, it's perfectly fine to bootstrap a
cluster that way, but an operating cluster should have at least two OSDs
running. See
http://ceph.com/docs/master/rados/operations/monitoring-osd-pg/#peering and
http://ceph.com/docs/master/rados/configuration/mon-osd-interaction/ to
learn how OSDs interact with each other and monitors.

Regards,


John



On Mon, May 6, 2013 at 8:04 AM, Guido Winkelmann 
guido-c...@thisisnotatest.de wrote:

 Am Montag, 6. Mai 2013, 16:59:12 schrieb Wido den Hollander:
  On 05/06/2013 04:51 PM, Guido Winkelmann wrote:
   Am Montag, 6. Mai 2013, 16:41:43 schrieb Wido den Hollander:
   On 05/06/2013 04:15 PM, Guido Winkelmann wrote:
   Am Montag, 6. Mai 2013, 16:05:31 schrieb Wido den Hollander:
   On 05/06/2013 04:00 PM, Guido Winkelmann wrote:
   Hi,
  
   How do I run a 1-node cluster with no replication?
  
   I'm trying to run a small 1-node cluster on my local workstation
 and
   another on my notebook for experimentation/development purposes,
 but
   since I only have on OSD, I'm always getting HEALTH_WARN as the
   cluster
   status from ceph -s. Can I somehow tell ceph to just not bother
 with
   replication for this cluster?
  
   Have you set min_size to 1 for all the pools?
  
   You mean in the crushmap?
  
   No, it's pool setting.
  
   See:
 http://ceph.com/docs/master/rados/operations/pools/#set-pool-values
  
   Hm, I set that to 1 now, and nothing changed:
  Have you also set size to 1? Meaning no replication.
 
  Both size and min_size should be set to 1.

 I set size to 1 now, too. ceph -s no longer reports degraded pgs now, but I
 still get a HEALTH_WARN:

 $ ceph -s
health HEALTH_WARN 384 pgs stuck unclean

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RAID 6 with Ceph 0.56.4

[John Wilkins]

Mark Kampe gave an excellent presentation on why Ceph may preclude the need
for RAID 6, and may provide you with better recovery advantages. Have a
look at it here: http://www.youtube.com/watch?v=La0Bxus6Fkg


On Sat, Apr 27, 2013 at 5:15 PM, ke_bac tinh
toiluonlatoi_852...@yahoo.comwrote:

 Hi all,

 I have 1 card raid on server, I use raid 6, then I divided into 4
 partitions , each partitions corresponding to 1 osd. I have 2 server == 8
 osd, but when I run ceph services, OSD frequently down.

 How can I make reasonable?

 Thanks
 Mr. Join's Pas
 *
 *

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] clean shutdown and failover of osd

[John Wilkins]

Will do.


On Sat, Apr 20, 2013 at 10:53 PM, Gregory Farnum g...@inktank.com wrote:

 On Sat, Apr 20, 2013 at 10:51 PM, James Harper
 james.har...@bendigoit.com.au wrote:
 
  [ This is a good query for ceph-users. ]
 
 
  Well... this is embarrassing. In reading the docs at
 http://ceph.com/docs/master/start/get-involved/ there was no mention of a
 users list so I just assumed there wasn't one. Looking again I see that if
 I go to the link from the main page
 http://ceph.com/resources/mailing-list-irc/ it's obviously there and I
 just never looked there.
 
  Sorry for the noise. Maybe the web/docs admin could add a link to the
 get-involved docs page?

 Haha, yep. Thanks for pointing that out — John? :)
 -Greg

 Software Engineer #42 @ http://inktank.com | http://ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph 0.56.4 - pgmap state: active+clean+scrubbing+deep

[John Wilkins]

This may be related to having your pool size = 1.  See
http://ceph.com/docs/master/rados/operations/troubleshooting-osd/#placement-groups-never-get-clean

Try setting your data size to 2: ceph osd pool set data size 2


On Mon, Apr 22, 2013 at 7:07 AM, MinhTien MinhTien 
tientienminh080...@gmail.com wrote:

 Dear all,

 - I use CentOS 6.3 up kernel 3.8.6-1.el6.elrepo.x86_64:  ceph storage
 (version 0.56.4), i set pool data (contains all data):

 ceph osd pool set data size 1

 - pool metadata:

 ceph osd pool set data size 2

 I have  osd, earch osd = 14TB (format ext4)

 I have 1 permanent error exists in the system.

 2013-04-22 20:24:20.942457 mon.0 [INF] pgmap v313221: 640 pgs: 638
 active+clean, 2 *active+clean+scrubbing+deep*; 17915 GB data, 17947 GB
 used, 86469 GB / 107 TB avail
 2013-04-22 20:24:12.256632 osd.1 [INF] 1.2e scrub ok
 2013-04-22 20:24:23.348560 mon.0 [INF] pgmap v313222: 640 pgs: 638
 active+clean, 2 *active+clean+scrubbing+deep*; 17915 GB data, 17947 GB
 used, 86469 GB / 107 TB avail
 2013-04-22 20:24:21.551528 osd.1 [INF] 1.3f scrub ok
 2013-04-22 20:24:52.009562 mon.0 [INF] pgmap v313223: 640 pgs: 638
 active+clean, 2 *active+clean+scrubbing+deep*; 17915 GB data, 17947 GB
 used, 86469 GB / 107 TB avail

 This makes me not access some data.

 I tried to restart, use command ceph pg repair  but error still exists

 I need some advice..

 Thanks



 --
 TienBM

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] has anyone successfully installed ceph with the crowbar

[John Wilkins]

Keep me posted on this, and I'll update the docs when we have a resolution.


On Thu, Apr 18, 2013 at 8:55 AM, Makkelie, R - SPLXL ramon.makke...@klm.com
 wrote:

 **
 well i tried to build the barclamp from
 https://github.com/ceph/barclamp-ceph
 and pacakge it with https://github.com/ceph/package-ceph-barclamp

 but the install fails

 so i also found a barclamp that is installing argonaut
 and it installs ceph

 but when i manually try to add a image in the volumes pool it fails.
 this is due to some permission failures probably because i try to follow a
 manual that is created for bobtail
 http://ceph.com/docs/master/rbd/rbd-openstack/




 On Thu, 2013-04-18 at 08:48 -0700, Gregory Farnum wrote:

 The barclamps were written against the crowbar Betty release, OpenStack
 Essex (which is the last one supported by Crowbar), and Ceph argonaut. JJ
 has updated them to use Bobtail, but I don't think anybody's run them
 against newer versions of Openstack. :(

  You should be able to find built versions of these on the Inktank or Ceph
 websites, though I don't remember where exactly.



  What are you trying to do, precisely?

  -Greg

 On Thursday, April 18, 2013, Makkelie, R - SPLXL wrote:

  Hi,

 Has anyone successfully installed Ceph using the ceph-barclamp with
 crowbar.
 if yes what version are you using and how did you created the barclamp
 and did you integrated it with Openstack folsom/Grizzly?

 GreetZ
 Ramonskie 
 For information, services and offers, please visit our web site:
 http://www.klm.com. This e-mail and any attachment may contain
 confidential and privileged material intended for the addressee only. If
 you are not the addressee, you are notified that no part of the e-mail or
 any attachment may be disclosed, copied or distributed, and that any other
 action related to this e-mail or attachment is strictly prohibited, and may
 be unlawful. If you have received this e-mail by error, please notify the
 sender immediately by return e-mail, and delete this message.

 Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its
 employees shall not be liable for the incorrect or incomplete transmission
 of this e-mail or any attachments, nor responsible for any delay in receipt.
 Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch
 Airlines) is registered in Amstelveen, The Netherlands, with registered
 number 33014286
 



 --
 Software Engineer #42 @ http://inktank.com | http://ceph.com


 
 For information, services and offers, please visit our web site:
 http://www.klm.com. This e-mail and any attachment may contain
 confidential and privileged material intended for the addressee only. If
 you are not the addressee, you are notified that no part of the e-mail or
 any attachment may be disclosed, copied or distributed, and that any other
 action related to this e-mail or attachment is strictly prohibited, and may
 be unlawful. If you have received this e-mail by error, please notify the
 sender immediately by return e-mail, and delete this message.

 Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its
 employees shall not be liable for the incorrect or incomplete transmission
 of this e-mail or any attachments, nor responsible for any delay in receipt.
 Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch
 Airlines) is registered in Amstelveen, The Netherlands, with registered
 number 33014286
 

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Health problem .. how to fix ?

[John Wilkins]

Stephane,

The monitoring section of operations explains what's happening, but I think
I probably need to do a better job of explaining unfound objects.
http://ceph.com/docs/master/rados/operations/monitoring-osd-pg/
http://ceph.com/docs/master/rados/operations/troubleshooting-osd/#unfound-objects

Let me know if those docs help, and let me know how I can improve on those
for you. That's an area that's not covered quite as well as it needs to be.


On Thu, Apr 18, 2013 at 8:26 AM, Stephane Boisvert 
stephane.boisv...@gameloft.com wrote:

  Hi,

   I configured a test 'cluster' and did play with it (moving osd folders
 around ie. journal file) and did break something. Now I think that this can
 occurs again  when we go prod. so I would like to know how I can fix it.. I
 don't care about loosing my files..

 Anyone can help? here's the logs


 HEALTH_WARN 15 pgs degraded; 1 pgs recovering; 15 pgs stale; 15 pgs stuck
 stale; 16 pgs stuck unclean; recovery 3/180 degraded (1.667%); 1/60 unfound
 (1.667%)
 pg 5.63 is stuck unclean for 157741.457285, current state
 stale+active+degraded, last acting [2]
 pg 4.66 is stuck unclean for 77312.285409, current state
 stale+active+degraded, last acting [2]
 pg 4.64 is stuck unclean for 157741.034570, current state
 stale+active+degraded, last acting [2]
 pg 5.65 is stuck unclean for 77312.285382, current state
 stale+active+degraded, last acting [2]
 pg 4.49 is stuck unclean for 77312.285021, current state
 stale+active+degraded, last acting [2]
 pg 5.48 is stuck unclean for 77312.285058, current state
 stale+active+degraded, last acting [2]
 pg 1.26 is stuck unclean for 77362.971821, current state
 active+recovering, last acting [5,2,1]
 pg 2.10 is stuck unclean for 157740.553908, current state
 stale+active+degraded, last acting [2]
 pg 4.e is stuck unclean for 157740.355222, current state
 stale+active+degraded, last acting [2]
 pg 5.d is stuck unclean for 157740.354260, current state
 stale+active+degraded, last acting [2]
 pg 5.0 is stuck unclean for 77312.264545, current state
 stale+active+degraded, last acting [2]
 pg 4.1 is stuck unclean for 77312.264416, current state
 stale+active+degraded, last acting [2]
 pg 3.2 is stuck unclean for 77312.263108, current state
 stale+active+degraded, last acting [2]
 pg 2.3 is stuck unclean for 77312.263026, current state
 stale+active+degraded, last acting [2]
 pg 4.71 is stuck unclean for 157740.352440, current state
 stale+active+degraded, last acting [2]
 pg 5.70 is stuck unclean for 157740.352547, current state
 stale+active+degraded, last acting [2]
 pg 5.63 is stuck stale for 77085.263183, current state
 stale+active+degraded, last acting [2]
 pg 4.66 is stuck stale for 77085.263186, current state
 stale+active+degraded, last acting [2]
 pg 4.64 is stuck stale for 77085.263187, current state
 stale+active+degraded, last acting [2]
 pg 5.65 is stuck stale for 77085.263191, current state
 stale+active+degraded, last acting [2]
 pg 4.49 is stuck stale for 77085.263186, current state
 stale+active+degraded, last acting [2]
 pg 5.48 is stuck stale for 77085.263191, current state
 stale+active+degraded, last acting [2]
 pg 2.10 is stuck stale for 77085.263258, current state
 stale+active+degraded, last acting [2]
 pg 4.e is stuck stale for 77085.263247, current state
 stale+active+degraded, last acting [2]
 pg 5.d is stuck stale for 77085.263245, current state
 stale+active+degraded, last acting [2]
 pg 5.0 is stuck stale for 77085.263241, current state
 stale+active+degraded, last acting [2]
 pg 4.1 is stuck stale for 77085.263245, current state
 stale+active+degraded, last acting [2]
 pg 3.2 is stuck stale for 77085.263242, current state
 stale+active+degraded, last acting [2]
 pg 2.3 is stuck stale for 77085.263247, current state
 stale+active+degraded, last acting [2]
 pg 4.71 is stuck stale for 77085.263239, current state
 stale+active+degraded, last acting [2]
 pg 5.70 is stuck stale for 77085.263245, current state
 stale+active+degraded, last acting [2]
 pg 1.26 is active+recovering, acting [5,2,1], 1 unfound
 recovery 3/180 degraded (1.667%); 1/60 unfound (1.667%)



 Thanks


 --








 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] No rolling updates from v0.56 to v0.60+?

[John Wilkins]

I believe that's the case. Joao spelled that out in his blog post here:
http://ceph.com/dev-notes/cephs-new-monitor-changes/

You can upgrade the monitors reasonably quickly though. Most clusters
operate with 3-5 monitors.


On Wed, Apr 17, 2013 at 7:40 AM, Guido Winkelmann
gu...@thisisnotatest.dewrote:

 Hi,

 I just tried upgrading parts of our experimental ceph cluster from 0.56.1
 to
 0.60, and it looks like the new mon-daemon from 0.60 cannot talk to those
 from
 0.56.1 at all.

 Long story short, we had to move some hardware around and during that time
 I
 had to shrink the cluster to one single machine. My plan was to expand it
 to
 three machines again, so that I would again have 3 mons and 3 osds, as
 before.
 I just installed the first new machine, going straight for 0.60, but
 leaving
 the remaining old one at 0.56.1. I added the new mon to the mon map
 according
 to the documentation and started the new mon daemon, but the mon-cluster
 wouldn't achieve quorum. In the logs for the new mon, I saw the following
 line
 repeated a lot:

 0 -- 10.6.224.129:6789/0  10.6.224.131:6789/0 pipe(0x2da5ec0 sd=20
 :37863
 s=1 pgs=0 cs=0 l=0).connect protocol version mismatch, my 10 != 9

 The old mon had no such lines in its log.

 I could only solve this by shutting down the old mon and upgrading it to
 0.60
 as well.

 It looks to me like this means rolling upgrades without downtime won't be
 possible from bobtail to cuttlefish. Is that correct?

 Guido
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Documentation Error in Adding/Removing OSDs

[John Wilkins]

Thanks Joe! I've made the change. You should see it up on the site shortly.


On Thu, Apr 11, 2013 at 10:00 AM, Joe Ryner jry...@cait.org wrote:

 Hi,

 I have found some issues in:
 http://ceph.com/docs/master/rados/operations/add-or-rm-osds

 In the adding section:
 Step 6 Should be ran before 1-5 as it outputs the OSD number when it
 exits.  I had a really hard time figuring this out.  I am currently running
 0.56.4 on RHEL 6.  The First 5 steps imply that you can pick an osd-number
 out of the either but really you have to use the osd number outputted by
 step 6.

 The following discussion help me figure this out.
 http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/11339


 Thanks,
 Joe
 --
 Joe Ryner
 Center for the Application of Information Technologies (CAIT)
 Production Coordinator
 P: (309) 298-1804
 F: (309) 298-2806
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Documentation Error in Adding/Removing OSDs

[John Wilkins]

Done. I also added some comments to the OSD configuration section noting
that OSDs names are numeric and incremental, e.g., 0,1,2,3; osd.0, osd.1,
etc.


On Thu, Apr 11, 2013 at 12:46 PM, Joe Ryner jry...@cait.org wrote:

 Probably should mention that the ceph osd create command will output
 what the new {osd-number} should be.

 Thanks for making the change so fast.

 Joe

 - Original Message -
 From: John Wilkins john.wilk...@inktank.com
 To: Joe Ryner jry...@cait.org
 Cc: ceph-users@lists.ceph.com
 Sent: Thursday, April 11, 2013 2:37:33 PM
 Subject: Re: [ceph-users] Documentation Error in Adding/Removing OSDs


 Thanks Joe! I've made the change. You should see it up on the site shortly.



 On Thu, Apr 11, 2013 at 10:00 AM, Joe Ryner  jry...@cait.org  wrote:


 Hi,

 I have found some issues in:
 http://ceph.com/docs/master/rados/operations/add-or-rm-osds

 In the adding section:
 Step 6 Should be ran before 1-5 as it outputs the OSD number when it
 exits. I had a really hard time figuring this out. I am currently running
 0.56.4 on RHEL 6. The First 5 steps imply that you can pick an osd-number
 out of the either but really you have to use the osd number outputted by
 step 6.

 The following discussion help me figure this out.
 http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/11339


 Thanks,
 Joe
 --
 Joe Ryner
 Center for the Application of Information Technologies (CAIT)
 Production Coordinator
 P: (309) 298-1804
 F: (309) 298-2806
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




 --
 John Wilkins
 Senior Technical Writer
 Intank
 john.wilk...@inktank.com
 (415) 425-9599
 http://inktank.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Newbie questions re: ceph setup

[John Wilkins]

I'll put in some notes about stepwise and starting from zero.


On Wed, Apr 3, 2013 at 9:58 AM, Gregory Farnum g...@inktank.com wrote:

 On Wed, Apr 3, 2013 at 9:45 AM, John Nielsen li...@jnielsen.net wrote:
  On Apr 1, 2013, at 3:33 PM, Gregory Farnum g...@inktank.com wrote:
 
  On Mon, Apr 1, 2013 at 2:16 PM, Sam Lang sl...@inktank.com wrote:
  On Mon, Apr 1, 2013 at 5:59 AM, Papaspyrou, Alexander
  papaspy...@adesso-mobile.de wrote:
  1. So far, I understand that OSD ids have to be numeric, nothing else
 in
  there. What I couldn't find is whether they really have to start at 0
 or 1,
  and whether I need to increase them stepwise? The background of my
 question
  is automation: It would make our life much simpler if I could use some
  calculated value for the OSD id (say, the node's decimal IP address
 and some
  local ID for the disk), because we could then statically assign them
 without
  having to know what other OSDs already exist.
 
  They need to start from 0.  The max_osd value maintained by the
  monitor bounds the number of possible osds, keeps the size of the
  osdmap down, etc.
 
  Note that using the ceph osd create command gives you an ID to use
  these days, and that's where you should be deriving these from.
  There's no support at all for making up your own OSD IDs; sorry.
 
  NOW you tell me.. :) I have a small cluster where I am using the first
 digit of the OSD to indicate which server it is in and the second to
 indicate which disk in that server it uses. Aside from occasional spurious
 entries (for the missing digits) in the CRUSH map it seems to work fine.
 
  Can you elaborate on why this might be Bad? I'm planning and beginning
 to deploy a larger cluster and was going to use four-digit OSD id's (two
 for the server, two for the disk). What problems should I expect if I do
 that?
 
  How hard would it be to support making up one's own OSD ID's?
 
  I would also suggest that this be better documented on the website. The
 5-minute quickstart says nothing about ID constraints and offers only 0 and
 1 as examples. The ceph-conf page says only that the instance ID for an
 OSD is always numeric. The add-or-rm-osds page mentions ceph osd
 create and its ability to set a UUID automatically, but only after
 several steps that require you to already know the osd-number.

 The problem is that these IDs are used as indexes into arrays, and
 people creating their own tends to lead to large and sparse arrays
 which actually become expensive to handle for a variety of reasons.
 Divorcing the name and id has been a thing in the back of our
 minds for a while but has never made it all the way to something we
 want to work on right now. :/ Until we support that we're pretty
 unlikely to support making up one's own IDs — we used to do so and it
 got people into so much trouble that we turned it off and just rely on
 monitor allocation at this point. It's possible the docs are out of
 date on this subject but they need not to be (John :).
 -Greg
 Software Engineer #42 @ http://inktank.com | http://ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Object location

[John Wilkins]

Sebastian,

You're correct. The usage has changed, so I've updated the doc to reflect
it. I ran through the procedure and it worked fine for me. Sorry Waed. I
hope that's all that was wrong.

On Wed, Mar 27, 2013 at 3:48 PM, Sebastien Han
sebastien@enovance.comwrote:

 Arf sorry, not 'odd' but 'osd' of course. (thanks autocompletion…)

 
 Sébastien Han
 Cloud Engineer

 Always give 100%. Unless you're giving blood.









 PHONE : +33 (0)1 49 70 99 72 – MOBILE : +33 (0)6 52 84 44 70
 EMAIL : sebastien@enovance.com – SKYPE : han.sbastien
 ADDRESS : 10, rue de la Victoire – 75009 Paris
 WEB : www.enovance.com – TWITTER : @enovance

 On Mar 27, 2013, at 11:36 PM, Sebastien Han sebastien@enovance.com
 wrote:

 Ok,

 I just noticed that the documentation seems to be wrong, the correct
 command to find the location of an object is:

 $ ceph odd map pool-name object-name

 Then, the error that you raised is pretty strange because even the object
 doesn't exist, the command will calculate the eventual location.

 Could you please paste _all_ the steps you've made to get this error?

 Thanks.

 
 Sébastien Han
 Cloud Engineer

 Always give 100%. Unless you're giving blood.



 image.png






 PHONE : +33 (0)1 49 70 99 72 – MOBILE : +33 (0)6 52 84 44 70
 EMAIL : sebastien@enovance.com – SKYPE : han.sbastien
 ADDRESS : 10, rue de la Victoire – 75009 Paris
 WEB : www.enovance.com – TWITTER : @enovance

 On Mar 26, 2013, at 1:36 PM, Waed Bataineh promiselad...@gmail.com
 wrote:

 Pool obj_name does not exist.


 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cluster Map Problems

[John Wilkins]

We need a bit more information. If you can do: ceph osd dump, ceph
osd tree, and paste your ceph conf, we might get a bit further. The
CRUSH hierarchy looks okay. I can't see the replica size from this
though.

Have you followed this procedure to see if your object is getting
remapped? 
http://ceph.com/docs/master/rados/operations/monitoring-osd-pg/#finding-an-object-location

On Thu, Mar 21, 2013 at 12:02 PM, Martin Mailand mar...@tuxadero.com wrote:
 Hi,

 I want to change my crushmap to reflect my setup, I have two racks with
 each 3 hosts. I want to use for the rbd pool a replication size of 2.
 The failure domain should be the rack, so each replica should be in each
 rack. That works so far.
 But if I shutdown a host the clusters stays degraded, but I want that
 the now missing replicas get replicated to the two remaining hosts in
 this rack.

 Here is crushmap.
 http://pastebin.com/UaB6LfKs

 Any idea what I did wrong?

 -martin
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com
(415) 425-9599
http://inktank.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com