Re: [ceph-users] Swift APIs not authenticating Rados gateway !!!
Hi Larry,
As you suggested I have changed to Ubuntu 10.04. Still I could not able to
figure out what is this problem. I skipped only two sections in ceph
documentation is 1) SSL 2) DNS , as I thought not needed any security to my
gateway.
1) I strongly have a doubt in specifying hostname in ceph.conf, rgw.conf.
Kindly share those files with me of your working setup.
2) How did you configure https link for rados gateway?
3) I have installed swift with sudo apt-get install swift. Is it ok?
Will be very thankful to you
Srinivas.
On Sat, Feb 22, 2014 at 3:03 AM, Liu, Larry wrote:
> Srinivasa, I pretty much think your problem is your fedora systems are
> missing some right lib files. I just got s3 working on my ubuntu raring
> setup. Just follow exactly what is written on
> http://ceph.com/docs/master/install/install-ceph-gateway/ . Still a
> question to everyone else: for swift API, what is the auth url? The
> command swift -A already works fine for me. Can't fine the swift
> auth url on the doc site.
>
> From: Srinivasa Rao Ragolu
> Date: Thursday, February 20, 2014 10:05 PM
> To: Microsoft Office User
> Cc: Yehuda Sadeh , "ceph-users@lists.ceph.com" <
> ceph-users@lists.ceph.com>
> Subject: Re: [ceph-users] Swift APIs not authenticating Rados gateway !!!
>
> Please help in making it easy Rados gateway configurable with Swift. It
> would be great support from you.
>
> I have skipped only two sections in
> http://ceph.com/docs/master/install/install-ceph-gateway/
>
> a) Enable SSL and b) Add wildcard to DNS
>
> Apart from these steps I have followed all other instruction on fedora
> 19..Please go through the attached configuration files.
>
> Still getting Authorisation failed : Http error 404
>
> Please help me.
>
> Srinivas.
>
>
>
> On Fri, Feb 21, 2014 at 1:06 AM, Liu, Larry wrote:
>
>> Hi Yehuda,
>>
>> Is there any doc on how to set the swift url (rgw swift url) configurable?
>>
>> On 2/19/14 7:42 AM, "Yehuda Sadeh" wrote:
>>
>> >On Wed, Feb 19, 2014 at 2:37 AM, Srinivasa Rao Ragolu
>> > wrote:
>> >> Hi all,
>> >>
>> >> I have setup cluster successfully and one node using to setup rados
>> >>gateway.
>> >> Machine is Fedora 19(all nodes)
>> >>
>> >> Steps I followed
>> >>
>> >> 1) Installed httpd, mod_fastcgi, ceph and ceph-radosgw using link
>> >> http://ceph.com/docs/master/install/install-ceph-gateway/
>> >>
>> >> Note : Did not follow "Enable SSL" and "Add wild card DNS" sections
>> >> 2) Made modifications in /etc/httpd/conf/httpd.conf,
>> >> /etc/httpd/conf.d/fastcgi.conf
>> >>
>> >> 3) Created rgw.conf in /etc/httpd/conf.d/
>> >>
>> >> 4) Followed the link
>> >> http://linuxmanpages.net/manpages/fedora19/man8/radosgw.8.html to
>> create
>> >> rgw.conf.
>> >>
>> >> 5) Added radosgw section in /etc/ceph/ceph.conf
>> >>
>> >> 6) Please see httpd.conf, fastcgi.conf, rgw.conf and ceph.conf as
>> >> attachments.
>> >>
>> >> 7) Now followed below steps
>> >>
>> >> a)
>> >>
>> >>corresponding radosgw script (/var/www/s3gw.fcgi):
>> >>
>> >>#!/bin/sh
>> >>exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n
>> >> client.radosgw.gateway
>> >>
>> >>
>> >> Gave execute permissions to s3gw.fcgi
>> >>
>> >>
>> >>
>> >>b)
>> >> ceph-authtool -C -n client.radosgw.gateway --gen-key
>> >> /etc/ceph/keyring.radosgw.gateway
>> >>ceph-authtool -n client.radosgw.gateway --cap mon 'allow r'
>> >>--cap osd
>> >> 'allow rwx' /etc/ceph/keyring.radosgw.gateway
>> >> ceph auth add client.radosgw.gateway --in-file=keyring.radosgw.gateway
>> >>
>> >>
>> >>
>> >> 8) sudo service ceph restart
>> >>
>> >>sudo service httpd restart
>> >>
>> >>sudo /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.rados.gateway
>> >>
>> >> 9) From the link http://ceph.com/docs/next/radosgw/config/, I
>> executed
>> >> "Create a gateway user", "Enabling swift access" sections
>> >>
>> >> 10) After above all steps if I run swift commands, I got following
>> error
>> >>
>> >> [gateway@gateway ceph]$ swift --verbose --debug -V 1.0 -A
>> >> http://radosgw.example.com/auth -U gateway:swift -K
>> >> "UoK\/XsR4zenFnq4z1sLIt6qSb7uPREfIEHtTG4JW" post test
>> >> ERROR:swiftclient:[Errno -2] Name or service not known
>> >> Traceback (most recent call last):
>> >> File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line
>> >>1101,
>> >> in _retry
>> >> self.url, self.token = self.get_auth()
>> >> File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line
>> >>1076,
>> >> in get_auth
>> >> insecure=self.insecure)
>> >> File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line
>> >>270,
>> >> in get_auth
>> >> kwargs.get('snet'))
>> >> File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line
>> >>182,
>> >> in get_auth_1_0
>> >> {'X-Auth-User': user, 'X-Auth-Key': key})
>> >> File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line
>> >>165,
>> >> in request_escaped
>> >> func(method, url, body=body, header
Re: [ceph-users] storage
Thanks Sebastien. -Original Message- From: Sebastien Han [mailto:sebastien@enovance.com] Sent: Tuesday, February 25, 2014 8:23 PM To: Gnan Kumar, Yalla Cc: ceph-users Subject: Re: [ceph-users] storage Hi, RBD blocks are stored as objects on a filesystem usually under: /var/lib/ceph/osd//current// RBD is just an abstraction layer. Cheers. Sébastien Han Cloud Engineer "Always give 100%. Unless you're giving blood." Phone: +33 (0)1 49 70 99 72 Mail: sebastien@enovance.com Address : 10, rue de la Victoire - 75009 Paris Web : www.enovance.com - Twitter : @enovance On 25 Feb 2014, at 13:09, yalla.gnan.ku...@accenture.com wrote: > Hi All, > > By default in which directory/directories, does ceph store the block device > files ? Is it in the /dev or other filesystem ? > > > Thanks > Kumar > > > This message is for the designated recipient only and may contain privileged, > proprietary, or otherwise confidential information. If you have received it > in error, please notify the sender immediately and delete the original. Any > other use of the e-mail by you is prohibited. Where allowed by local law, > electronic communications with Accenture and its affiliates, including e-mail > and instant messaging (including content), may be scanned by our systems for > the purposes of information security and assessment of internal compliance > with Accenture policy. . > __ > > > www.accenture.com > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] PG folder hierarchy
Got it. Thanks Greg for the response! Thanks, Guang On Feb 26, 2014, at 11:51 AM, Gregory Farnum wrote: > On Tue, Feb 25, 2014 at 7:13 PM, Guang wrote: >> Hello, >> Most recently when looking at PG's folder splitting, I found that there was >> only one sub folder in the top 3 / 4 levels and start having 16 sub folders >> starting from level 6, what is the design consideration behind this? >> >> For example, if the PG root folder is '3.1905_head', in the first level, it >> only has one sub folder 'DIR_5' and then one sub folder 'DIR_0', and then >> 'DIR_9', under which there are two sub folders 'DIR_1' and 'DIR_9', starting >> from which, the next level has 16 sub folders. >> >> If we start splitting into 16 sub folders in the very first level, we may >> potential gain better performance with less dentry lookup (though most >> likely the root level been cached). > > It's an implementation detail of the FileStore (the part of the OSD > that stores data in the filesystem). Each of those folders represents > an ever-smaller division of the hash space that objects live in. The > more PGs you have, the less hash space each one covers, so there's > that trail of folders. > It's a bit unfortunate, because as you mention it involves more > metadata memory caching, but fixing it would require some fairly > detailed code in a critical path. The cost of fixing it and the risk > of breaking things haven't been worth it yet. > -Greg > Software Engineer #42 @ http://inktank.com | http://ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] PG folder hierarchy
On Tue, Feb 25, 2014 at 7:13 PM, Guang wrote: > Hello, > Most recently when looking at PG's folder splitting, I found that there was > only one sub folder in the top 3 / 4 levels and start having 16 sub folders > starting from level 6, what is the design consideration behind this? > > For example, if the PG root folder is '3.1905_head', in the first level, it > only has one sub folder 'DIR_5' and then one sub folder 'DIR_0', and then > 'DIR_9', under which there are two sub folders 'DIR_1' and 'DIR_9', starting > from which, the next level has 16 sub folders. > > If we start splitting into 16 sub folders in the very first level, we may > potential gain better performance with less dentry lookup (though most > likely the root level been cached). It's an implementation detail of the FileStore (the part of the OSD that stores data in the filesystem). Each of those folders represents an ever-smaller division of the hash space that objects live in. The more PGs you have, the less hash space each one covers, so there's that trail of folders. It's a bit unfortunate, because as you mention it involves more metadata memory caching, but fixing it would require some fairly detailed code in a critical path. The cost of fixing it and the risk of breaking things haven't been worth it yet. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] PG folder hierarchy
Hello, Most recently when looking at PG’s folder splitting, I found that there was only one sub folder in the top 3 / 4 levels and start having 16 sub folders starting from level 6, what is the design consideration behind this? For example, if the PG root folder is ‘3.1905_head’, in the first level, it only has one sub folder ‘DIR_5’ and then one sub folder ‘DIR_0’, and then ‘DIR_9’, under which there are two sub folders ‘DIR_1’ and ‘DIR_9’, starting from which, the next level has 16 sub folders. If we start splitting into 16 sub folders in the very first level, we may potential gain better performance with less dentry lookup (though most likely the root level been cached). Thanks, Guang___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Ubuntu 13.10 packages
Hi All, Just wondering if there was a reason for no packages for Ubuntu Saucy in http://ceph.com/packages/ceph-extras/debian/dists/. Could do with upgrading to fix a few bugs but would hate to have to drop Ceph from being handled through the package manager! Thanks, -Michael ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] CDS Giant Schedule
Greetings! Just wanted to let people know that the schedule has been published for the next Ceph Developer Summit (March 04-05, 2014): https://wiki.ceph.com/Planning/CDS/CDS_Giant_(Mar_2014) There may still be a few last minute tweaks, but for the most part that should be what we're working with next week. I wanted to get it out in plenty of time so that people can make arrangements and/or plan their sessions a bit. Let me apologize to anyone who is going to experience a bit of sleep deprivation. There were a ton of great submissions and we are still experimenting with finding a "sweet spot" as far as times. I think east coasters on the APAC-friendly day got the worst of it at just after 1am. In any case, schedule is up and linked to blueprints and pads. Session owners if you could toss a rough outline of what you would like to discuss into the pad I think that would help ensure that each session is smooth and productive. If anyone has questions or concerns feel free to send them my way. Thanks! Best Regards, Patrick McGarry Director, Community || Inktank http://ceph.com || http://inktank.com @scuttlemonkey || @ceph || @inktank ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] pages stuck unclean (but remapped)
With the reweight-by-utilization applied, CRUSH is failing to generate
mappings of enough OSDs, so the system is falling back to keeping
around copies that already exist, even though they aren't located on
the correct CRUSH-mapped OSDs (since there aren't enough OSDs).
Are your OSDs correctly weighted in CRUSH by their size? If not, you
want to apply that there and return all of the monitor override
weights to 1.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
On Tue, Feb 25, 2014 at 9:19 AM, Gautam Saxena wrote:
> So the "backfill_tooful" was an old state; it disappeared after I
> reweighted. Yesterday, I even set up the Ceph system's tunables to optimal,
> added one more osd, let it rebalance, and then after rebalancing, I ran a
> "ceph osd reweight-by-utilization 105". After several hours, though, CEPH
> stabilized (that is no more recovery), but the final state is worse than
> before. So here are my questions (I also included the results of "ceph -s"
> right after these questions):
>
> 1) why are 153 pages in "active+remapped" but not going anywhere? Shouldn't
> they be more like "active+remapped+wait_backfill" instead?
> 2) Why are 10 pages "active+remapped+backfilling" but there is no actual
> activity occurring in CEPH? Shouldn't it instead say
> "active+remapped+wait_backfill+backfill_toofull"
> 3) Why is there a backfill_tooful when all my osds are well under 95% full
> -- in fact, they are all under 81% full (as determined by "df -h" command?)
> (One theory I have is that the "too_full" percentage is based NOT on the
> actual physical space on the OSD, but on the *reweighted* physical space. Is
> this theory accurate?
> 4) When I did a "ceph pg dump", I saw that all 153 pages that are in
> active+remapped have only 1 OSD in the "up" state but 2 OSDs in the "acting"
> state. I'm confused as to the difference between "up" and "acting" -- does
> this scenario mean that if I lose 1 OSD that in the "up" state, I lose data
> for that page? Or does the "acting" mean that the page data is still on 2
> OSDs, so I can afford to lose 1 OSD.
>
> --> ceph -s produces:
>
>
> [root@ia2 ceph]# ceph -s
> cluster 14f78538-6085-43f9-ac80-e886ca4de119
> health HEALTH_WARN 10 pgs backfill; 5 pgs backfill_toofull; 10 pgs
> backfilling; 173 pgs stuck unclean; recovery 44940/5858368 objects degraded
> (0.767%)
> monmap e9: 3 mons at
> {ia1=192.168.1.11:6789/0,ia2=192.168.1.12:6789/0,ia3=192.168.1.13:6789/0},
> election epoch 500, quorum 0,1,2 ia1,ia2,ia3
> osdmap e9700: 23 osds: 23 up, 23 in
> pgmap v2003396: 1500 pgs, 1 pools, 11225 GB data, 2841 kobjects
> 22452 GB used, 23014 GB / 45467 GB avail
> 44940/5858368 objects degraded (0.767%)
> 1327 active+clean
>5 active+remapped+wait_backfill
>5 active+remapped+wait_backfill+backfill_toofull
> 153 active+remapped
> 10 active+remapped+backfilling
> client io 4369 kB/s rd, 64377 B/s wr, 26 op/s
> ==
>
>
>
> On Sun, Feb 23, 2014 at 8:09 PM, Gautam Saxena wrote:
>>
>> I have 19 pages that are stuck unclean (see below result of ceph -s). This
>> occurred after I executed a "ceph osd reweight-by-utilization 108" to
>> resolve problems with "backfill_too_full" messages, which I believe occurred
>> because my OSDs sizes vary significantly in size (from a low of 600GB to a
>> high of 3 TB). How can I get ceph to get these pages out of stuck-unclean?
>> (And why is this occurring anyways?) My best guess of how to fix (though I
>> don't know why) is that I need to run:
>>
>> ceph osd crush tunables optimal.
>>
>> However, my kernel version (on a fully up-to-date Centos 6.5) is 2.6.32,
>> which is well below the minimum required version of 3.6 that's stated in the
>> documentation (http://ceph.com/docs/master/rados/operations/crush-map/) --
>> so if I must run "ceph osd crush tunables optimal" to fix this problem, I
>> presume I must upgrade my kernel first, right?...Any thoughts or am I
>> chasing the wrong solution -- I want to avoid kernel upgrade unless it's
>> needed.)
>>
>> =
>>
>> [root@ia2 ceph4]# ceph -s
>> cluster 14f78538-6085-43f9-ac80-e886ca4de119
>> health HEALTH_WARN 19 pgs backfilling; 19 pgs stuck unclean; recovery
>> 42959/5511127 objects degraded (0.779%)
>> monmap e9: 3 mons at
>> {ia1=192.168.1.11:6789/0,ia2=192.168.1.12:6789/0,ia3=192.168.1.13:6789/0},
>> election epoch 496, quorum 0,1,2 ia1,ia2,ia3
>> osdmap e7931: 23 osds: 23 up, 23 in
>> pgmap v1904820: 1500 pgs, 1 pools, 10531 GB data, 2670 kobjects
>> 18708 GB used, 26758 GB / 45467 GB avail
>> 42959/5511127 objects degraded (0.779%)
>> 1481 active+clean
>> 19 active+remapped+backfilling
>> client io 1457 B/s wr, 0 op/s
>>
>> [root@ia2 ceph4]# ceph -v
>> ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60)
Re: [ceph-users] questions about monitor data and ceph recovery
Hi! > 2. One node (with 8 osds) goes offline. Will ceph automatically replicate all > objects on the remaining node to maintain number of replicas = 2? > No, because it can no longer satisfy your CRUSH rules. Your crush rule states > 1x copy pr. node and it will keep it that way. The cluster will go into a > degraded state until you can bring up another node (ie all your data now is > very vulnerable). I think it is often suggested to run with 3x replica if > possible - or at the very least nr_nodes = replicas + 1. If you had to make > it replicate on the remaining node you'd have to change your CRUSH rule to > replicate based on OSD and not node. But then you'll most likely have > problems when 1 node dies because objects could easily be on 2x OSD on the > failed node. Is it possible to define "fallback" crush rule, which must work if main rule cannot obtain needed number of replicas? Pavel. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Upgrading ceph
25 февр. 2014 г., в 14:13, Srinivasa Rao Ragolu написал(а):
> always better to have same version in all the nodes of cluster to avoid
> integration issues rule out.
But, while updating, some nodes will run on older version for a some period. Is
this ok?
Pavel.
> On Tue, Feb 25, 2014 at 2:40 PM, Pavel V. Kaygorodov wrote:
> Hi!
>
> Is it possible to have monitors and osd daemons running different versions of
> ceph in one cluster?
>
> Pavel.
>
>
>
>
> 25 февр. 2014 г., в 10:56, Srinivasa Rao Ragolu
> написал(а):
>
> > Yes Sahana,
> >
> > First of all uninstall ceph packages from your node.
> > then
> > Approach for rpm based:
> > You just open /etc/yum.repos.d/ceph.repo
> > Replace the {ceph-stable-release} with emperor and {distro} with rpm based
> > distro
> > baseurl=http://ceph.com/rpm-{ceph-stable-release}/{distro}/noarch
> >
> >
> > Now:
> > sudo yum update; sudo yum install {ceph-***whatever**}
> >
> >
> > Thanks,
> > Srinivas.
> >
> >
> >
> >
> >
> > On Tue, Feb 25, 2014 at 11:56 AM, Sahana wrote:
> > Hi ,
> >
> > May I know how to upgrade ceph from one version to another(Eg : Dumpling to
> > emperor). Is there any specific steps ?
> >
> > Thanks
> > Sahana
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] pages stuck unclean (but remapped)
So the "backfill_tooful" was an old state; it disappeared after I
reweighted. Yesterday, I even set up the Ceph system's tunables to optimal,
added one more osd, let it rebalance, and then after rebalancing, I ran a
"ceph osd reweight-by-utilization 105". After several hours, though, CEPH
stabilized (that is no more recovery), but the final state is worse than
before. So here are my questions (I also included the results of "ceph -s"
right after these questions):
1) why are 153 pages in "active+remapped" but not going anywhere? Shouldn't
they be more like "active+remapped+wait_backfill" instead?
2) Why are 10 pages "active+remapped+backfilling" but there is no actual
activity occurring in CEPH? Shouldn't it instead say
"active+remapped+wait_backfill+backfill_toofull"
3) Why is there a backfill_tooful when all my osds are well under 95% full
-- in fact, they are all under 81% full (as determined by "df -h" command?)
(One theory I have is that the "too_full" percentage is based NOT on the
actual physical space on the OSD, but on the *reweighted* physical space.
Is this theory accurate?
4) When I did a "ceph pg dump", I saw that all 153 pages that are in
active+remapped have only 1 OSD in the "up" state but 2 OSDs in the
"acting" state. I'm confused as to the difference between "up" and "acting"
-- does this scenario mean that if I lose 1 OSD that in the "up" state, I
lose data for that page? Or does the "acting" mean that the page data is
still on 2 OSDs, so I can afford to lose 1 OSD.
--> ceph -s produces:
[root@ia2 ceph]# ceph -s
cluster 14f78538-6085-43f9-ac80-e886ca4de119
health HEALTH_WARN 10 pgs backfill; 5 pgs backfill_toofull; 10 pgs
backfilling; 173 pgs stuck unclean; recovery 44940/5858368 objects degraded
(0.767%)
monmap e9: 3 mons at {ia1=
192.168.1.11:6789/0,ia2=192.168.1.12:6789/0,ia3=192.168.1.13:6789/0},
election epoch 500, quorum 0,1,2 ia1,ia2,ia3
osdmap e9700: 23 osds: 23 up, 23 in
pgmap v2003396: 1500 pgs, 1 pools, 11225 GB data, 2841 kobjects
22452 GB used, 23014 GB / 45467 GB avail
44940/5858368 objects degraded (0.767%)
1327 active+clean
5 active+remapped+wait_backfill
5 active+remapped+wait_backfill+backfill_toofull
153 active+remapped
10 active+remapped+backfilling
client io 4369 kB/s rd, 64377 B/s wr, 26 op/s
==
On Sun, Feb 23, 2014 at 8:09 PM, Gautam Saxena wrote:
> I have 19 pages that are stuck unclean (see below result of ceph -s). This
> occurred after I executed a "ceph osd reweight-by-utilization 108" to
> resolve problems with "backfill_too_full" messages, which I believe
> occurred because my OSDs sizes vary significantly in size (from a low of
> 600GB to a high of 3 TB). How can I get ceph to get these pages out of
> stuck-unclean? (And why is this occurring anyways?) My best guess of how to
> fix (though I don't know why) is that I need to run:
>
> ceph osd crush tunables optimal.
>
> However, my kernel version (on a fully up-to-date Centos 6.5) is 2.6.32,
> which is well below the minimum required version of 3.6 that's stated in
> the documentation (http://ceph.com/docs/master/rados/operations/crush-map/
> ) -- so if I must run "ceph osd crush tunables optimal" to fix this
> problem, I presume I must upgrade my kernel first, right?...Any thoughts or
> am I chasing the wrong solution -- I want to avoid kernel upgrade unless
> it's needed.)
>
> =
>
> [root@ia2 ceph4]# ceph -s
> cluster 14f78538-6085-43f9-ac80-e886ca4de119
> health HEALTH_WARN 19 pgs backfilling; 19 pgs stuck unclean; recovery
> 42959/5511127 objects degraded (0.779%)
> monmap e9: 3 mons at {ia1=
> 192.168.1.11:6789/0,ia2=192.168.1.12:6789/0,ia3=192.168.1.13:6789/0},
> election epoch 496, quorum 0,1,2 ia1,ia2,ia3
> osdmap e7931: 23 osds: 23 up, 23 in
> pgmap v1904820: 1500 pgs, 1 pools, 10531 GB data, 2670 kobjects
> 18708 GB used, 26758 GB / 45467 GB avail
> 42959/5511127 objects degraded (0.779%)
> 1481 active+clean
> 19 active+remapped+backfilling
> client io 1457 B/s wr, 0 op/s
>
> [root@ia2 ceph4]# ceph -v
> ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60)
>
> [root@ia2 ceph4]# uname -r
> 2.6.32-431.3.1.el6.x86_64
>
>
>
--
*Gautam Saxena *
President & CEO
Integrated Analysis Inc.
Making Sense of Data.(tm)
Biomarker Discovery Software | Bioinformatics Services | Data Warehouse
Consulting | Data Migration Consulting
www.i-a-inc.com
[ceph-users] Fscache and Ceph
After reading the following on Fscahce integration with CephFS, I'm would like to know which version of Linux kernel has all the Fscache patches available? http://ceph.com/community/first-impressions-through-fscache-and-ceph/ Do we know when these patches will be available in future release of Ubuntu or RHEL (CentOS)? Thanks! Amit Amit Vijairania | 415.610.9908 --*-- ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Most stable version of ceph?
Greetings, I've been running dumpling for several months and it seems very stable. I'm about to spin up a new ceph environment. Would I be advised to install emperor? Or, since dumpling is solid, just stick with it? Thanks much, JR ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] goofy results for df
On Mon, Feb 24, 2014 at 11:48 PM, Markus Goldberg
wrote:
> Hi Gregory,
> here we go:
>
> root@bd-a:/mnt/myceph#
> root@bd-a:/mnt/myceph# ls -la
> insgesamt 4
> drwxr-xr-x 1 root root 25928099891213 Feb 24 14:14 .
> drwxr-xr-x 4 root root 4096 Aug 30 10:34 ..
> drwx-- 1 root root 25920394954765 Feb 7 10:07 Backup
> drwxr-xr-x 1 root root32826961870 Feb 24 14:51 temp
>
> I think, the big numbers above are the used bytes consumed in the directory
Yep, those are the "recursive statistics" on directory size, and it
agrees with your du.
>
> root@bd-a:/mnt/myceph#
> root@bd-a:/mnt/myceph# ceph osd dump
> epoch 146
> fsid ad1a4f5c-cc86-4fef-b8f6-
> created 2014-02-03 10:13:55.109549
> modified 2014-02-17 10:37:41.750786
> flags
>
> pool 0 'data' replicated size 3 min_size 2 crush_ruleset 0 object_hash
> rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 flags hashpspool
> crash_replay_interval 45
> pool 1 'metadata' replicated size 3 min_size 2 crush_ruleset 0 object_hash
> rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 flags hashpspool
> pool 2 'rbd' replicated size 3 min_size 2 crush_ruleset 0 object_hash
> rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 flags hashpspool
> pool 3 'markus' replicated size 3 min_size 2 crush_ruleset 0 object_hash
> rjenkins pg_num 100 pgp_num 100 last_change 15 owner 0 flags hashpspool
> pool 4 'ecki' replicated size 3 min_size 2 crush_ruleset 0 object_hash
> rjenkins pg_num 100 pgp_num 100 last_change 17 owner 0 flags hashpspool
> pool 5 'kevin' replicated size 3 min_size 2 crush_ruleset 0 object_hash
> rjenkins pg_num 100 pgp_num 100 last_change 19 owner 0 flags hashpspool
> pool 6 'alfresco' replicated size 3 min_size 2 crush_ruleset 0 object_hash
> rjenkins pg_num 100 pgp_num 100 last_change 21 owner 0 flags hashpspool
> pool 7 'bacula' replicated size 3 min_size 2 crush_ruleset 0 object_hash
> rjenkins pg_num 100 pgp_num 100 last_change 23 owner 0 flags hashpspool
> pool 8 'bareos' replicated size 3 min_size 2 crush_ruleset 0 object_hash
> rjenkins pg_num 100 pgp_num 100 last_change 25 owner 0 flags hashpspool
> pool 9 'bs3' replicated size 3 min_size 2 crush_ruleset 0 object_hash
> rjenkins pg_num 100 pgp_num 100 last_change 27 owner 0 flags hashpspool
> pool 10 'Verw-vdc2' replicated size 3 min_size 2 crush_ruleset 0 object_hash
> rjenkins pg_num 100 pgp_num 100 last_change 54 owner 0 flags hashpspool
>
> max_osd 3
> osd.0 up in weight 1 up_from 139 up_thru 143 down_at 138
> last_clean_interval [134,135) xxx.xxx.xxx.xx0:6801/2105
> 192.168.1.20:6800/2105 192.168.1.20:6801/2105 xxx.xxx.xxx.xx0:6802/2105
> exists,up b2b1a1bd-f6ba-47f2-8485-xx7e
> osd.1 up in weight 1 up_from 143 up_thru 143 down_at 142
> last_clean_interval [120,135) xxx.xxx.xxx.xx1:6801/2129
> 192.168.1.21:6800/2129 192.168.1.21:6801/2129 xxx.xxx.xxx.xx1:6802/2129
> exists,up 2dc1dd2c-ce99-4e7d-9672-xxx.xxx.xxx.xx1f
> osd.2 up in weight 1 up_from 139 up_thru 143 down_at 138
> last_clean_interval [125,135) xxx.xxx.xxx.xx2:6801/2018
> 192.168.1.22:6800/2018 192.168.1.22:6801/2018 xxx.xxx.xxx.xx2:6802/2018
> exists,up 83d293a1-5f34-4086-a3d6-xxx.xxx.xxx.xx7c
>
>
> root@bd-a:/mnt/myceph#
> root@bd-a:/mnt/myceph# ceph -s
> cluster ad1a4f5c-cc86-4fef-b8f6-
> health HEALTH_OK
> monmap e1: 3 mons at
> {bd-0=xxx.xxx.xxx.xx0:6789/0,bd-1=xxx.xxx.xxx.xx1:6789/0,bd-2=xxx.xxx.xxx.xx2:6789/0},
> election epoch 506, quorum 0,1,2 bd-0,bd-1,bd-2
> mdsmap e171: 1/1/1 up {0=bd-2=up:active}, 2 up:standby
> osdmap e146: 3 osds: 3 up, 3 in
> pgmap v81525: 992 pgs, 11 pools, 31456 MB data, 8058 objects
> 94792 MB used, 61309 GB / 61408 GB avail
> 992 active+clean
But this indicates that raw RADOS indeed believes that it only has
~30GB of data total, which isn't enough to store 21TB of filesystem
data! Are the available sizes correct? Can you dump the pgmap and
paste bin it somewhere we can look at (sorry, I meant that rather than
the OSDMap to begin with; my bad!)? I'm wondering if the stats are
corrupted or what.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] CephFS and slow requests
Okay, well, let's try and track some of these down. What's the content of the "ceph.layout" xattr on the directory you're running this test in? Can you verify that pool 0 is the data pool used by CephFS, and that all reported slow ops are in that pool? Can you record the IO patterns on an OSD while this test is being run and see what it does? (I'm wondering if none of the CephFS pools are in the page cache due to lack of use, and it's seeking all over trying to find them once the test starts.) -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com On Mon, Feb 24, 2014 at 11:54 PM, Dan van der Ster wrote: > It's really bizarre, since we can easily pump ~1GB/s into the cluster with > rados bench from a single 10Gig-E client. We only observe this with kernel > CephFS on that host -- which is why our original theory something like this: >- client caches 4GB of writes >- client starts many opening IOs in parallel to flush that cache >- each individual 4MB write is taking longer than 30s to send from the > client to the OSD, due to the 1Gig-E network interface on the client. > > But in that we assume quite a lot about the implementations of librados and > the osd. But something like this would also explain why only the cephfs > writes are becoming slow -- the 2kHz of other (mostly RBD) IOs are not > affected by this "overload". > > Cheers, Dan ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Size of objects in Ceph
> > You can't change it afterwards, but when creating and image you can > supply the --order value and change the default 22 into something you > like: > > 22 = 4096KB > 23 = 8192KB > 24 = 16384KB > 25 = 32768KB > 26 = 65536KB > >> Or is it a fixed value in Ceph architecture ? >> > > No, you can set it per RBD image. I have to see, for most situations > the default 4MB works just fine. > Ok very good, thank you Wido ! And what are the steps to do that in Swift RadosGW when creating objects ? And for CephFS for each file (is it per file, per directory...) ? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Size of objects in Ceph
On 02/25/2014 03:54 PM, Florent Bautista wrote: Hi all, I'm new with Ceph and I would like to know if there is any way of changing size of Ceph's internal objects. I mean, when I put an image on RBD for exemple, I can see this: rbd -p CephTest info base-127-disk-1 rbd image 'base-127-disk-1': size 32768 MB in 8192 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.347c274b0dc51 format: 2 features: layering "4096 kB objects" => how can I change size of objects ? You can't change it afterwards, but when creating and image you can supply the --order value and change the default 22 into something you like: 22 = 4096KB 23 = 8192KB 24 = 16384KB 25 = 32768KB 26 = 65536KB Or is it a fixed value in Ceph architecture ? No, you can set it per RBD image. I have to see, for most situations the default 4MB works just fine. Thank you ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Wido den Hollander 42on B.V. Phone: +31 (0)20 700 9902 Skype: contact42on ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Size of objects in Ceph
Hi, The value can be set during the image creation. Start with this: http://ceph.com/docs/master/man/8/rbd/#striping Followed by the example section. Sébastien Han Cloud Engineer "Always give 100%. Unless you're giving blood.” Phone: +33 (0)1 49 70 99 72 Mail: sebastien@enovance.com Address : 10, rue de la Victoire - 75009 Paris Web : www.enovance.com - Twitter : @enovance On 25 Feb 2014, at 15:54, Florent Bautista wrote: > Hi all, > > I'm new with Ceph and I would like to know if there is any way of > changing size of Ceph's internal objects. > > I mean, when I put an image on RBD for exemple, I can see this: > > rbd -p CephTest info base-127-disk-1 > rbd image 'base-127-disk-1': >size 32768 MB in 8192 objects >order 22 (4096 kB objects) >block_name_prefix: rbd_data.347c274b0dc51 >format: 2 >features: layering > > > "4096 kB objects" => how can I change size of objects ? > > Or is it a fixed value in Ceph architecture ? > > Thank you > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com signature.asc Description: Message signed with OpenPGP using GPGMail ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Size of objects in Ceph
Hi all, I'm new with Ceph and I would like to know if there is any way of changing size of Ceph's internal objects. I mean, when I put an image on RBD for exemple, I can see this: rbd -p CephTest info base-127-disk-1 rbd image 'base-127-disk-1': size 32768 MB in 8192 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.347c274b0dc51 format: 2 features: layering "4096 kB objects" => how can I change size of objects ? Or is it a fixed value in Ceph architecture ? Thank you ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] storage
Hi, RBD blocks are stored as objects on a filesystem usually under: /var/lib/ceph/osd//current// RBD is just an abstraction layer. Cheers. Sébastien Han Cloud Engineer "Always give 100%. Unless you're giving blood.” Phone: +33 (0)1 49 70 99 72 Mail: sebastien@enovance.com Address : 10, rue de la Victoire - 75009 Paris Web : www.enovance.com - Twitter : @enovance On 25 Feb 2014, at 13:09, yalla.gnan.ku...@accenture.com wrote: > Hi All, > > By default in which directory/directories, does ceph store the block device > files ? Is it in the /dev or other filesystem ? > > > Thanks > Kumar > > > This message is for the designated recipient only and may contain privileged, > proprietary, or otherwise confidential information. If you have received it > in error, please notify the sender immediately and delete the original. Any > other use of the e-mail by you is prohibited. Where allowed by local law, > electronic communications with Accenture and its affiliates, including e-mail > and instant messaging (including content), may be scanned by our systems for > the purposes of information security and assessment of internal compliance > with Accenture policy. . > __ > > www.accenture.com > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com signature.asc Description: Message signed with OpenPGP using GPGMail ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] how to set public IP in ceph.conf?
Hi all, I hit the same problem here when adding new monitors: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-October/005483.html I understand that I should set up the public address in ceph.conf. But I am really confused about the public network setting in the doc. Where should I set the address in ceph.conf? Is it on my ceph admin node? To my understanding the public ip address should be different for all ceph nodes. If I set it on my admin node then all the nodes get the same public IP address. True? BTW, initially I set the public IP in ceph.conf. Then the system can not find keyrings and I got stuck there. After I remove the public address, at least I can have 1 monitor and 2 osds working properly. Say I have two network interfaces for each node: host internal IP external IP Node2 192.168.0.2 202.197.0.2 Node3 192.168.0.3 202.197.0.3 Node4 192.168.0.4 202.197.0.4 Of course, the external IPs are connected to the outside through NAT. But they can reach the outside. What should be included in my ceph.conf? Thanks! Bill Chang ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to Configure Cinder to access multiple pools
Hi, Please have a look at the cinder multi-backend functionality: examples here: http://www.sebastien-han.fr/blog/2013/04/25/ceph-and-cinder-multi-backend/ Cheers. Sébastien Han Cloud Engineer "Always give 100%. Unless you're giving blood.” Phone: +33 (0)1 49 70 99 72 Mail: sebastien@enovance.com Address : 10, rue de la Victoire - 75009 Paris Web : www.enovance.com - Twitter : @enovance On 25 Feb 2014, at 14:42, Vikrant Verma wrote: > Hi All, > > I am using cinder as a front end for volume storage in Openstack > configuration. > Ceph is used as storage back-end. > > Currently cinder uses only one pool (in my case pool name is "volumes" ) for > its volume storage. > I want cinder to use multiple ceph pools for volume storage > > > --following is the cinder.conf--- > volume_driver=cinder.volume.drivers.rbd.RBDDriver > rbd_pool=volumes > rbd_ceph_conf=/etc/ceph/ceph.conf > rbd_flatten_volume_from_snapshot=false > > > Please let me know if it is possible to have multiple pools associated to > cinder, let me know how to configure it. > > Regards, > Vikrant > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com signature.asc Description: Message signed with OpenPGP using GPGMail ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Fwd: Re: ceph-deploy stuck, unable to find keyring
THanks Srinivasa. The instruction is not for Ubuntu but I ll take it as
ref.
I actually found the problem is due to the public IP setting in ceph.conf
on my admin node. I remove that line and the problem is fixed and the
cluster can be set up. But a new problem occurs that I can only have one
monitor. If adding more that won't work due to the lack of public network
setting.
On Tue, Feb 25, 2014 at 1:32 PM, Srinivasa Rao Ragolu wrote:
> Please cross verify by following blog written very detailed manner.
>
> http://karan-mj.blogspot.in/2013/12/ceph-storage-part-2.html
> and
> http://karan-mj.blogspot.in/2013/12/ceph-installation-part-2.html
>
> It will definitely help you in resolving the issue. Please follow every
> step mentioned on your nodes.
>
> be in /etc/ceph/conf directory and go to super user mode
>
> Thanks,
> Srinivas
>
>
> On Tue, Feb 25, 2014 at 6:55 AM, 常乐 wrote:
>
>> -- Forwarded message --
>> From: "常乐"
>> Date: Feb 25, 2014 8:31 AM
>> Subject: Re: [ceph-users] ceph-deploy stuck, unable to find keyring
>> To: "Alfredo Deza"
>> Cc:
>>
>> hi Alfredo,
>>
>> I tried manual before but get stuck creating osd. Now i decide to fix all
>> these problems. I google d for these problems but there seems to be no
>> sign on the internet. I also attached the log here. I didnt see any sign
>> that these keys are created.
>> I am using ceph 0.72, and ubuntu servet lts 12.04.
>>
>> Btw, do i have to do extra work than in the ceph doc? Could you pls
>> provide some refs? Thanks!
>>
>>
>> [ceph_deploy.cli][INFO ] Invoked (1.3.5): /usr/bin/ceph-deploy mon
>> create-initial[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts
>> compute1
>>
>> [ceph_deploy.mon][DEBUG ] detecting platform for host compute1 ...
>>
>> [compute1][DEBUG ] connected to host: compute1
>>
>> [compute1][DEBUG ] detect platform information from remote host
>>
>> [compute1][DEBUG ] detect machine type[ceph_deploy.mon][INFO ] distro
>> info: Ubuntu 12.04 precise
>>
>> [compute1][DEBUG ] determining if provided host has same hostname in
>> remote
>>
>> [compute1][DEBUG ] get remote short hostname[compute1][DEBUG ] deploying
>> mon to compute1
>>
>> [compute1][DEBUG ] get remote short hostname
>>
>> [compute1][DEBUG ] remote hostname: compute1
>>
>> [compute1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
>>
>> [compute1][DEBUG ] create the mon path if it does not exist
>>
>> [compute1][DEBUG ] checking for done path:
>> /var/lib/ceph/mon/ceph-compute1/done[compute1][DEBUG ] done path does not
>> exist: /var/lib/ceph/mon/ceph-compute1/done[compute1][INFO ] creating
>> keyring file: /var/lib/ceph/tmp/ceph-compute1.mon.keyring
>>
>> [compute1][DEBUG ] create the monitor keyring file[compute1][INFO ]
>> Running command: sudo ceph-mon --cluster ceph --mkfs -i compute1 --keyring
>> /var/lib/ceph/tmp/ceph-compute1.mon.keyring[compute1][DEBUG ] ceph-mon: set
>> fsid to 42b3c138-b0fb-49a9-b9c4-13a7841fe68b[compute1][DEBUG ] ceph-mon:
>> created monfs at /var/lib/ceph/mon/ceph-compute1 for
>> mon.compute1[compute1][INFO ] unlinking keyring file
>> /var/lib/ceph/tmp/ceph-compute1.mon.keyring
>>
>> [compute1][DEBUG ] create a done file to avoid re-doing the mon deployment
>>
>> [compute1][DEBUG ] create the init path if it does not exist
>>
>> [compute1][DEBUG ] locating the `service` executable...[compute1][INFO ]
>> Running command: sudo initctl emit ceph-mon cluster=ceph id=compute1
>>
>> [compute1][INFO ] unlinking keyring file
>> /var/lib/ceph/tmp/ceph-compute1.mon.keyring
>>
>> [compute1][DEBUG ] create a done file to avoid re-doing the mon deployment
>>
>> [compute1][DEBUG ] create the init path if it does not exist
>>
>> [compute1][DEBUG ] locating the `service` executable...
>>
>> [compute1][INFO ] Running command: sudo initctl emit ceph-mon
>> cluster=ceph id=compute1[compute1][INFO ] Running command: sudo ceph
>> --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.compute1.asok
>> mon_status[compute1][DEBUG ]
>>
>>
>> [compute1][DEBUG ] status for monitor: mon.compute1
>>
>> [compute1][DEBUG ] {[compute1][DEBUG ] "election_epoch": 2,
>>
>> [compute1][DEBUG ] "extra_probe_peers": [[compute1][DEBUG ] "
>> 192.168.2.23:6789/0"
>>
>> [compute1][DEBUG ] ],
>>
>> [compute1][DEBUG ] "monmap": {[compute1][DEBUG ] "created":
>> "0.00",
>>
>> [compute1][DEBUG ] "epoch": 1,
>>
>> [compute1][DEBUG ] "fsid":
>> "42b3c138-b0fb-49a9-b9c4-13a7841fe68b",[compute1][DEBUG ] "modified":
>> "0.00",
>>
>> [compute1][DEBUG ] "mons": [
>>
>>
>> 550,1 92%
>>
>> [compute1][DEBUG ] create the monitor keyring file
>>
>> [compute1][INFO ] Running command: sudo ceph-mon --cluster ceph --mkfs
>> -i compute1 --keyring /var/lib/ceph/tmp/ceph-compute1.mon.keyring
>>
>> [compute1][DEBUG ] ceph-mon: set fsid to
>> 42b3c138-b0fb-49a9-b9c4-13a7841fe68b
>>
>> [compute1][DEBUG ] cep
Re: [ceph-users] ceph-deploy stuck, unable to find keyring
Thanks Altredo. 'ceph-deploy purge NODE &&
ceph-deploy purge-data' really helps.
On Tue, Feb 25, 2014 at 9:24 PM, Alfredo Deza wrote:
> To start over is easier to do it from scratch to avoid configuration
> problems and other things that might bite you back
> like stale keyrings.
>
> It has happened to me more than once :)
>
> When I need to start from scratch I call 'ceph-deploy purge NODE &&
> ceph-deploy purge-data'
>
> And then remove all the files in the current working directory and
> then start again.
>
> On Mon, Feb 24, 2014 at 7:31 PM, 常乐 wrote:
> > hi Alfredo,
> >
> > I tried manual before but get stuck creating osd. Now i decide to fix all
> > these problems. I google d for these problems but there seems to be no
> sign
> > on the internet. I also attached the log here. I didnt see any sign that
> > these keys are created.
> > I am using ceph 0.72, and ubuntu servet lts 12.04.
> >
> > Btw, do i have to do extra work than in the ceph doc? Could you pls
> provide
> > some refs? Thanks!
> >
> >
> > [ceph_deploy.cli][INFO ] Invoked (1.3.5): /usr/bin/ceph-deploy mon
> > create-initial[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts
> > compute1
> >
> > [ceph_deploy.mon][DEBUG ] detecting platform for host compute1 ...
> >
> > [compute1][DEBUG ] connected to host: compute1
> >
> > [compute1][DEBUG ] detect platform information from remote host
> >
> > [compute1][DEBUG ] detect machine type[ceph_deploy.mon][INFO ] distro
> info:
> > Ubuntu 12.04 precise
> >
> > [compute1][DEBUG ] determining if provided host has same hostname in
> remote
> >
> > [compute1][DEBUG ] get remote short hostname[compute1][DEBUG ] deploying
> mon
> > to compute1
> >
> > [compute1][DEBUG ] get remote short hostname
> >
> > [compute1][DEBUG ] remote hostname: compute1
> >
> > [compute1][DEBUG ] write cluster configuration to
> /etc/ceph/{cluster}.conf
> >
> > [compute1][DEBUG ] create the mon path if it does not exist
> >
> > [compute1][DEBUG ] checking for done path:
> > /var/lib/ceph/mon/ceph-compute1/done[compute1][DEBUG ] done path does not
> > exist: /var/lib/ceph/mon/ceph-compute1/done[compute1][INFO ] creating
> > keyring file: /var/lib/ceph/tmp/ceph-compute1.mon.keyring
> >
> > [compute1][DEBUG ] create the monitor keyring file[compute1][INFO ]
> Running
> > command: sudo ceph-mon --cluster ceph --mkfs -i compute1 --keyring
> > /var/lib/ceph/tmp/ceph-compute1.mon.keyring[compute1][DEBUG ] ceph-mon:
> set
> > fsid to 42b3c138-b0fb-49a9-b9c4-13a7841fe68b[compute1][DEBUG ] ceph-mon:
> > created monfs at /var/lib/ceph/mon/ceph-compute1 for
> > mon.compute1[compute1][INFO ] unlinking keyring file
> > /var/lib/ceph/tmp/ceph-compute1.mon.keyring
> >
> > [compute1][DEBUG ] create a done file to avoid re-doing the mon
> deployment
> >
> > [compute1][DEBUG ] create the init path if it does not exist
> >
> > [compute1][DEBUG ] locating the `service` executable...[compute1][INFO ]
> > Running command: sudo initctl emit ceph-mon cluster=ceph id=compute1
> >
> > [compute1][INFO ] unlinking keyring file
> > /var/lib/ceph/tmp/ceph-compute1.mon.keyring
> >
> > [compute1][DEBUG ] create a done file to avoid re-doing the mon
> deployment
> >
> > [compute1][DEBUG ] create the init path if it does not exist
> >
> > [compute1][DEBUG ] locating the `service` executable...
> >
> > [compute1][INFO ] Running command: sudo initctl emit ceph-mon
> cluster=ceph
> > id=compute1[compute1][INFO ] Running command: sudo ceph --cluster=ceph
> > --admin-daemon /var/run/ceph/ceph-mon.compute1.asok
> > mon_status[compute1][DEBUG ]
> >
>
> >
> > [compute1][DEBUG ] status for monitor: mon.compute1
> >
> > [compute1][DEBUG ] {[compute1][DEBUG ] "election_epoch": 2,
> >
> > [compute1][DEBUG ] "extra_probe_peers": [[compute1][DEBUG ]
> > "192.168.2.23:6789/0"
> >
> > [compute1][DEBUG ] ],
> >
> > [compute1][DEBUG ] "monmap": {[compute1][DEBUG ] "created":
> > "0.00",
> >
> > [compute1][DEBUG ] "epoch": 1,
> >
> > [compute1][DEBUG ] "fsid":
> > "42b3c138-b0fb-49a9-b9c4-13a7841fe68b",[compute1][DEBUG ] "modified":
> > "0.00",
> >
> > [compute1][DEBUG ] "mons": [
> >
> > 550,1
> > 92%
> >
> > [compute1][DEBUG ] create the monitor keyring file
> >
> > [compute1][INFO ] Running command: sudo ceph-mon --cluster ceph --mkfs
> -i
> > compute1 --keyring /var/lib/ceph/tmp/ceph-compute1.mon.keyring
> >
> > [compute1][DEBUG ] ceph-mon: set fsid to
> > 42b3c138-b0fb-49a9-b9c4-13a7841fe68b
> >
> > [compute1][DEBUG ] ceph-mon: created monfs at
> > /var/lib/ceph/mon/ceph-compute1 for mon.compute1
> >
> > [compute1][INFO ] unlinking keyring file
> > /var/lib/ceph/tmp/ceph-compute1.mon.keyring
> >
> > [compute1][DEBUG ] create a done file to avoid re-doing the mon
> deployment
> >
> > [compute1][DEBUG ] create the init path if it does not exist
> >
> > [compute1][DEBUG ] locat
[ceph-users] How to Configure Cinder to access multiple pools
Hi All, I am using cinder as a front end for volume storage in Openstack configuration. Ceph is used as storage back-end. Currently cinder uses only one pool (in my case pool name is "volumes" ) for its volume storage. I want cinder to use multiple ceph pools for volume storage --following is the cinder.conf--- volume_driver=cinder.volume.drivers.rbd.RBDDriver *rbd_pool=volumes* rbd_ceph_conf=/etc/ceph/ceph.conf rbd_flatten_volume_from_snapshot=false Please let me know if it is possible to have multiple pools associated to cinder, let me know how to configure it. Regards, Vikrant ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph-deploy stuck, unable to find keyring
To start over is easier to do it from scratch to avoid configuration
problems and other things that might bite you back
like stale keyrings.
It has happened to me more than once :)
When I need to start from scratch I call 'ceph-deploy purge NODE &&
ceph-deploy purge-data'
And then remove all the files in the current working directory and
then start again.
On Mon, Feb 24, 2014 at 7:31 PM, 常乐 wrote:
> hi Alfredo,
>
> I tried manual before but get stuck creating osd. Now i decide to fix all
> these problems. I google d for these problems but there seems to be no sign
> on the internet. I also attached the log here. I didnt see any sign that
> these keys are created.
> I am using ceph 0.72, and ubuntu servet lts 12.04.
>
> Btw, do i have to do extra work than in the ceph doc? Could you pls provide
> some refs? Thanks!
>
>
> [ceph_deploy.cli][INFO ] Invoked (1.3.5): /usr/bin/ceph-deploy mon
> create-initial[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts
> compute1
>
> [ceph_deploy.mon][DEBUG ] detecting platform for host compute1 ...
>
> [compute1][DEBUG ] connected to host: compute1
>
> [compute1][DEBUG ] detect platform information from remote host
>
> [compute1][DEBUG ] detect machine type[ceph_deploy.mon][INFO ] distro info:
> Ubuntu 12.04 precise
>
> [compute1][DEBUG ] determining if provided host has same hostname in remote
>
> [compute1][DEBUG ] get remote short hostname[compute1][DEBUG ] deploying mon
> to compute1
>
> [compute1][DEBUG ] get remote short hostname
>
> [compute1][DEBUG ] remote hostname: compute1
>
> [compute1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
>
> [compute1][DEBUG ] create the mon path if it does not exist
>
> [compute1][DEBUG ] checking for done path:
> /var/lib/ceph/mon/ceph-compute1/done[compute1][DEBUG ] done path does not
> exist: /var/lib/ceph/mon/ceph-compute1/done[compute1][INFO ] creating
> keyring file: /var/lib/ceph/tmp/ceph-compute1.mon.keyring
>
> [compute1][DEBUG ] create the monitor keyring file[compute1][INFO ] Running
> command: sudo ceph-mon --cluster ceph --mkfs -i compute1 --keyring
> /var/lib/ceph/tmp/ceph-compute1.mon.keyring[compute1][DEBUG ] ceph-mon: set
> fsid to 42b3c138-b0fb-49a9-b9c4-13a7841fe68b[compute1][DEBUG ] ceph-mon:
> created monfs at /var/lib/ceph/mon/ceph-compute1 for
> mon.compute1[compute1][INFO ] unlinking keyring file
> /var/lib/ceph/tmp/ceph-compute1.mon.keyring
>
> [compute1][DEBUG ] create a done file to avoid re-doing the mon deployment
>
> [compute1][DEBUG ] create the init path if it does not exist
>
> [compute1][DEBUG ] locating the `service` executable...[compute1][INFO ]
> Running command: sudo initctl emit ceph-mon cluster=ceph id=compute1
>
> [compute1][INFO ] unlinking keyring file
> /var/lib/ceph/tmp/ceph-compute1.mon.keyring
>
> [compute1][DEBUG ] create a done file to avoid re-doing the mon deployment
>
> [compute1][DEBUG ] create the init path if it does not exist
>
> [compute1][DEBUG ] locating the `service` executable...
>
> [compute1][INFO ] Running command: sudo initctl emit ceph-mon cluster=ceph
> id=compute1[compute1][INFO ] Running command: sudo ceph --cluster=ceph
> --admin-daemon /var/run/ceph/ceph-mon.compute1.asok
> mon_status[compute1][DEBUG ]
>
>
> [compute1][DEBUG ] status for monitor: mon.compute1
>
> [compute1][DEBUG ] {[compute1][DEBUG ] "election_epoch": 2,
>
> [compute1][DEBUG ] "extra_probe_peers": [[compute1][DEBUG ]
> "192.168.2.23:6789/0"
>
> [compute1][DEBUG ] ],
>
> [compute1][DEBUG ] "monmap": {[compute1][DEBUG ] "created":
> "0.00",
>
> [compute1][DEBUG ] "epoch": 1,
>
> [compute1][DEBUG ] "fsid":
> "42b3c138-b0fb-49a9-b9c4-13a7841fe68b",[compute1][DEBUG ] "modified":
> "0.00",
>
> [compute1][DEBUG ] "mons": [
>
> 550,1
> 92%
>
> [compute1][DEBUG ] create the monitor keyring file
>
> [compute1][INFO ] Running command: sudo ceph-mon --cluster ceph --mkfs -i
> compute1 --keyring /var/lib/ceph/tmp/ceph-compute1.mon.keyring
>
> [compute1][DEBUG ] ceph-mon: set fsid to
> 42b3c138-b0fb-49a9-b9c4-13a7841fe68b
>
> [compute1][DEBUG ] ceph-mon: created monfs at
> /var/lib/ceph/mon/ceph-compute1 for mon.compute1
>
> [compute1][INFO ] unlinking keyring file
> /var/lib/ceph/tmp/ceph-compute1.mon.keyring
>
> [compute1][DEBUG ] create a done file to avoid re-doing the mon deployment
>
> [compute1][DEBUG ] create the init path if it does not exist
>
> [compute1][DEBUG ] locating the `service` executable...
>
> [compute1][INFO ] Running command: sudo initctl emit ceph-mon cluster=ceph
> id=compute1
>
> [compute1][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon
> /var/run/ceph/ceph-mon.compute1.asok mon_status
>
> [compute1][DEBUG ]
>
>
> [compute1][DEBUG ] status for monitor: mon.compute1
[ceph-users] storage
Hi All, By default in which directory/directories, does ceph store the block device files ? Is it in the /dev or other filesystem ? Thanks Kumar This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. . __ www.accenture.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] questions about monitor data and ceph recovery
Hi Pavel, Will try and answer some of your questions: My first question will be about monitor data directory. How much space I > need to reserve for it? Can monitor-fs be corrupted if monitor goes out of > storage space? > We have about 20GB partitions for monitors - they really don't use much space, but in case you need to do some extra logging it is nice to have (and ceph doing max debug consumes scary amounts of space). Also if you look in the monitor log they constantly monitor for free space. I don't know what will happen if a monitor runs full (or close to full), but I'm guessing that monitor will simply be marked as down or stopped somehow. You can change some of the values for a mon about how much data to keep before trimming etc. > > I also have questions about ceph auto-recovery process. > For example, I have two nodes with 8 drives on each, each drive is > presented as separate osd. The number of replicas = 2. I have wrote a crush > ruleset, which picks two nodes and one osd on each to store replicas. Which > will happens on following scenarios: > > 1. One drive in one node failed. Will ceph automatically re-replicate > affected objects? Where replicas will be stored? > Yes, as long as you have available space on the node that lost one OSD the data that was on that disk will be distributed aross the remaining 7 OSD on that node (according to your CRUSH rules) > > 1.1 The failed osd will appears online again with all of it's data. How > ceph cluster will deal with it? > This is just how I _think_ it works; please correct me if I'm wrong. All OSD have an internal map (pg map) which is constantly updated throughout the cluster. When any OSD goes offline/down and is started back up the latest pgmap of that OSD is 'diffed' up vs the latest map from the cluster and then the cluster can generate a new map based on what it has/had, what is missing/updated and generate a new map with the objects the newly started OSD should have. Then it will start to replicate and only get the changed/new objects. Bottom line, this really just works and works very well. > > 2. One node (with 8 osds) goes offline. Will ceph automatically replicate > all objects on the remaining node to maintain number of replicas = 2? > No, because it can no longer satisfy your CRUSH rules. Your crush rule states 1x copy pr. node and it will keep it that way. The cluster will go into a degraded state until you can bring up another node (ie all your data now is very vulnerable). I think it is often suggested to run with 3x replica if possible - or at the very least nr_nodes = replicas + 1. If you had to make it replicate on the remaining node you'd have to change your CRUSH rule to replicate based on OSD and not node. But then you'll most likely have problems when 1 node dies because objects could easily be on 2x OSD on the failed node. > > 2.1 The failed node goes online again with all data. How ceph cluster will > deal with it? > Same as the above with the OSD. Cheers, Martin > Thanks in advance, > Pavel. > > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
