[ceph-users] rbd format 2 stripe-count != 1 cannot be mapped with rbd.ko kernel 3.13.5
OK, it seems during the rbd creation with --stripe-count != 1 you have to follow the rule: stripe-unit * stripe-count = object-size So, EINVAL below was correct, but the message librbd: STRIPINGV2 and format 2 or later required for non-default striping is wrong and confusing. Now I have the problme that even with rbd.ko from kernel 3.13.5 this volume is not mountable: [root@rx37-0 ~]# rbd create --pool SSD-r2 --size 20480 --order 22 --image-format 2 --stripe-unit 1048576 --stripe-count 4 t2-1 [root@rx37-0 ~]# rbd -p SSD-r2 info t2-1 rbd image 't2-1': size 20480 MB in 5120 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.12722ae8944a format: 2 features: layering, striping stripe unit: 1024 kB stripe count: 4 [root@rx37-0 ~]# rbd -p SSD-r2 map t2-1 rbd: add failed: (22) Invalid argument [root@rx37-0 ~]# uname -a Linux rx37-0.primary.ceph-poc.fsc.net 3.13.5-1.el6.elrepo.x86_64 #1 SMP Sat Feb 22 21:34:07 EST 2014 x86_64 x86_64 x86_64 GNU/Linux [root@rx37-0 ~]# dmesg | tail rbd: image t2-1: unsupported stripe unit (got 1048576 want 4194304) Any hints ? Regards, -Dieter On Tue, Mar 11, 2014 at 04:37:46PM +0100, Jean-Charles LOPEZ wrote: Hi Greg, but our default also has stripe-count = 1 so that no more than 1 stripe-unit is included in each order x object. So if you do --order 16?stripe-unit 65536 ?stripe-count 1 it then works I?m not sure if this is what you meant. JC On Mar 11, 2014, at 08:32, Gregory Farnum g...@inktank.com wrote: If the stripe size and object size are the same it's just chunking -- that's our default. Should work fine. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com On Tue, Mar 11, 2014 at 8:23 AM, Jean-Charles LOPEZ jc.lo...@inktank.com wrote: Hi Dieter, you have a problem with your command. You set order = 16 so your RBD objects is going to be 65536 bytes Then you tell RBD that you stripe-unit is going to be 65536 which is the size of your full object. Either decrease the size of --stripe-unit to 8192 for example Or increase order so that it is bigger than your stripe unit and contains a multiple of stripe-units (e.g. 21) And it will work without any problem JC On Mar 11, 2014, at 07:22, Kasper Dieter dieter.kas...@ts.fujitsu.com wrote: So, should I open a bug report ? STRIPINGV2 feature was added in Ceph v0.53, and I'm running v0.61 and using '--image-format 2' during 'rbd create' Regards, -Dieter On Tue, Mar 11, 2014 at 03:13:28PM +0100, Srinivasa Rao Ragolu wrote: of course. rbd userland utilities provide you create images on RADOS as block storage. On Tue, Mar 11, 2014 at 7:37 PM, Kasper Dieter [1]dieter.kas...@ts.fujitsu.com wrote: I know, that format2 in rbd.ko is supported with kernel version 3.10 and above. But, if I want to create an rbd-image only the Ceph Userland services should be involved, shouldn't it ? -Dieter BTW the kernel version on the nodes hosting the OSDs processes is 2.6.32-358.el6.x86_64 but I can also boot with a 3.10.32 kernel. On Tue, Mar 11, 2014 at 02:57:05PM +0100, Srinivasa Rao Ragolu wrote: Please check the kernel version . Only kernel version 3.10 and above are supported to create format type 2 images. On Tue, Mar 11, 2014 at 7:16 PM, Kasper Dieter [1][2]dieter.kas...@ts.fujitsu.com wrote: When using rbd create ... --image-format 2 in some cases this CMD is rejected by EINVAL with the message librbd: STRIPINGV2 and format 2 or later required for non-default striping But, in v0.61.9 STRIPINGV2 and format 2 should be supported [root@rx37-3 ~]# rbd create --pool SSD-r2 --size 20480 --order 16 --image-format 2 --stripe-unit 65536 --stripe-count 4 t2 rbd: create error: (22) Invalid argument 2014-03-11 14:39:03.885185 7f15bc170760 -1 librbd: STRIPINGV2 and format 2 or later required for non-default striping [root@rx37-3 ~]# ceph -v ceph version 0.61.9 (7440dcd135750839fa0f00263f80722ff6f51e90) Any hints ? Regards, -Dieter ___ ceph-users mailing list [2][3]ceph-users@lists.ceph.com [3][4]http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com References Visible links 1. mailto:[5]dieter.kas...@ts.fujitsu.com 2. mailto:[6]ceph-users@lists.ceph.com 3. [7]http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com References Visible links 1. mailto:dieter.kas...@ts.fujitsu.com 2. mailto:dieter.kas...@ts.fujitsu.com 3. mailto:ceph-users@lists.ceph.com 4. http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 5. mailto:dieter.kas...@ts.fujitsu.com 6. mailto:ceph-users@lists.ceph.com 7.
Re: [ceph-users] how to configure ceph object gateway
Hi, what commands are “not found”? This page for configuring the RGW works fine as far as I know as I used it no later than a week ago. Can you please give us more details? What is your layout (radosgw installed on a ceph node, mon node, standalone node)? Note: In order to get it running, remember you need to have a web server installed and running (apache), ceph base packages obviously, swift if you want to use the swift tool, s3cmd also, s3curl, … JC On Mar 10, 2014, at 19:35, wsnote wsn...@163.com wrote: OS: CentOS 6.4 version: ceph 0.67.7 Hello, everyone. With the help of document, I have install ceph gateway. But I don't know how to configure it. The web http://ceph.com/docs/master/radosgw/config/ has many command not found.I thought it's written in the ubuntu. can anyone help? Thanks! ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] How to use ceph
Hi, My name is Ashraful Arefeen. I want to use ceph for testing purpose. Is it possible to use it in a single machine (I mean in one computer)? If it is possible then what will be the preferable configuration of the computer and in that case what are required software apart from ceph? I have started according to the documentation. But there I found that it needs different host computers to setup nodes using ceph-deploy command. Moreover, the documentation is for production purpose, I want to use it for testing and relatively lower configuration. -- Ashraful Arefeen Software Engineer, Therap Services, LLC. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] rbd create ... STRIPINGV2 and format 2 or later required
Hi Dieter, you have a problem with your command. You set order = 16 so your RBD objects is going to be 65536 bytes Then you tell RBD that you stripe-unit is going to be 65536 which is the size of your full object. Either decrease the size of —stripe-unit to 8192 for example Or increase order so that it is bigger than your stripe unit and contains a multiple of stripe-units (e.g. 21) And it will work without any problem JC On Mar 11, 2014, at 07:22, Kasper Dieter dieter.kas...@ts.fujitsu.com wrote: So, should I open a bug report ? STRIPINGV2 feature was added in Ceph v0.53, and I'm running v0.61 and using '--image-format 2' during 'rbd create' Regards, -Dieter On Tue, Mar 11, 2014 at 03:13:28PM +0100, Srinivasa Rao Ragolu wrote: of course. rbd userland utilities provide you create images on RADOS as block storage. On Tue, Mar 11, 2014 at 7:37 PM, Kasper Dieter [1]dieter.kas...@ts.fujitsu.com wrote: I know, that format2 in rbd.ko is supported with kernel version 3.10 and above. But, if I want to create an rbd-image only the Ceph Userland services should be involved, shouldn't it ? -Dieter BTW the kernel version on the nodes hosting the OSDs processes is 2.6.32-358.el6.x86_64 but I can also boot with a 3.10.32 kernel. On Tue, Mar 11, 2014 at 02:57:05PM +0100, Srinivasa Rao Ragolu wrote: Please check the kernel version . Only kernel version 3.10 and above are supported to create format type 2 images. On Tue, Mar 11, 2014 at 7:16 PM, Kasper Dieter [1][2]dieter.kas...@ts.fujitsu.com wrote: When using rbd create ... --image-format 2 in some cases this CMD is rejected by EINVAL with the message librbd: STRIPINGV2 and format 2 or later required for non-default striping But, in v0.61.9 STRIPINGV2 and format 2 should be supported [root@rx37-3 ~]# rbd create --pool SSD-r2 --size 20480 --order 16 --image-format 2 --stripe-unit 65536 --stripe-count 4 t2 rbd: create error: (22) Invalid argument 2014-03-11 14:39:03.885185 7f15bc170760 -1 librbd: STRIPINGV2 and format 2 or later required for non-default striping [root@rx37-3 ~]# ceph -v ceph version 0.61.9 (7440dcd135750839fa0f00263f80722ff6f51e90) Any hints ? Regards, -Dieter ___ ceph-users mailing list [2][3]ceph-users@lists.ceph.com [3][4]http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com References Visible links 1. mailto:[5]dieter.kas...@ts.fujitsu.com 2. mailto:[6]ceph-users@lists.ceph.com 3. [7]http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com References Visible links 1. mailto:dieter.kas...@ts.fujitsu.com 2. mailto:dieter.kas...@ts.fujitsu.com 3. mailto:ceph-users@lists.ceph.com 4. http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 5. mailto:dieter.kas...@ts.fujitsu.com 6. mailto:ceph-users@lists.ceph.com 7. http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Some Questions about using ceph with VMware
Hello everyone, this is the first time i ever write to a mailing list, please be patient with me (especially for my poor english)… Im trying to reach my Bachelors Degree in Computer Science, Im doing a Project which involves ceph. I’am able to setup a ceph Cluster, but there are a few things i can’t Figure out… As I’am setting up the cluster with virtual machines im facing a little Problem: clock skew: every time i reboot on node a clock skew is detected. If i restart the Monitor on which the clock skew is detected the Problem is gone. But this is not what i want to show in my presentation. I already enabled the VMware tools, but the Problem persists. Does it make more sense to enable NTP? Are there any performance Issues to expect if i use a drive with ceph-disk-prepare (or activate) /dev/sdb directly ? Are there any (big) performance Issues to expect using these virtual drives instead of „real“ drives? Kind regards Florian ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] rbd format 2 stripe-count != 1 cannot be mapped with rbd.ko kernel 3.13.5
Please see this Email on ceph-devel ---snip--- Date: Thu, 15 Aug 2013 14:30:24 +0200 From: Damien Churchill dam...@gmail.com To: Kasper, Dieter dieter.kas...@ts.fujitsu.com CC: ceph-de...@vger.kernel.org ceph-de...@vger.kernel.org Subject: Re: rbd: format 2 support in rbd.ko ? On 15 August 2013 12:42, Kasper Dieter dieter.kas...@ts.fujitsu.com wrote: When will 'format 2' support be available in rbd.ko ? Format 2 is supported in kernel 3.10 or newer. ---snip--- This was also the statement from Sage on the Ceph day in Frankfurt two weeks ago (as far as I understood) -Dieter krbd doesn't support STRIPINGV2 yet, so that EINVAL is correct. The reason you can 'rbd map' a --image-format 2 image in general is that v2 with default striping parameters behaves exactly the same as v1. Thanks, Ilya On Wed, Mar 12, 2014 at 02:33:21PM +0100, Ilya Dryomov wrote: On Wed, Mar 12, 2014 at 2:02 PM, Kasper, Dieter dieter.kas...@ts.fujitsu.com wrote: No difference: [root@rx37-0 ~]# rbd map --pool SSD-r2 t2-1 rbd: add failed: (22) Invalid argument [root@rx37-0 ~]# dmesg | tail (...) rbd: image t2-1: unsupported stripe unit (got 1048576 want 4194304) The strace of 'rbd map' says: open(/sys/bus/rbd/add, O_WRONLY) = 3 write(3, 192.168.113.13:6789,192.168.113.14:6789,192.168.113.15:6789 name=admin,key=client.admin SSD-r2 t2-1, 99) = -1 EINVAL (Invalid argument) ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Some Questions about using ceph with VMware
On Wednesday, March 12, 2014, Florian Krauß bruder1...@googlemail.com wrote: Hello everyone, this is the first time i ever write to a mailing list, please be patient with me (especially for my poor english)... Im trying to reach my Bachelors Degree in Computer Science, Im doing a Project which involves ceph. I'am able to setup a ceph Cluster, but there are a few things i can't Figure out... As I'am setting up the cluster with virtual machines im facing a little Problem: clock skew: every time i reboot on node a clock skew is detected. If i restart the Monitor on which the clock skew is detected the Problem is gone. But this is not what i want to show in my presentation. I already enabled the VMware tools, but the Problem persists. Does it make more sense to enable NTP? It sounds like maybe you're using VM hibernation or something. Are there any performance Issues to expect if i use a drive with ceph-disk-prepare (or activate) /dev/sdb directly ? That should be fine. Are there any (big) performance Issues to expect using these virtual drives instead of real drives? Well, each of those drives are going to see frequent syncs, so it'll be slow. If it turns out too badly you could set the VM to not pass those kinds of data safety primitives to the real hardware. -Greg Kind regards Florian ___ ceph-users mailing list ceph-users@lists.ceph.com javascript:; http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Software Engineer #42 @ http://inktank.com | http://ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] PG Scaling
Correction: Sorry min_size is at 1 everywhere. Thank you. Karol Kozubal From: Karol Kozubal karol.kozu...@elits.commailto:karol.kozu...@elits.com Date: Wednesday, March 12, 2014 at 12:06 PM To: ceph-users@lists.ceph.commailto:ceph-users@lists.ceph.com ceph-users@lists.ceph.commailto:ceph-users@lists.ceph.com Subject: PG Scaling Hi Everyone, I am deploying an openstack deployment with Fuel 4.1 and have a 20 node ceph deployment of c6220’s with 3 osd’s and 1 journaling disk per node. When first deployed each storage pool is configured with the correct size and min_size attributes however fuel doesn’t seem to apply the correct number of pg’s to the pools based on the number of osd’s that we actually have. I make the adjustments using the following (20 nodes * 3 OSDs)*100 / 3 replicas = 2000 ceph osd pool volumes set size 3 ceph osd pool volumes set min_size 3 ceph osd pool volumes set pg_num 2000 ceph osd pool volumes set pgp_num 2000 ceph osd pool images set size 3 ceph osd pool images set min_size 3 ceph osd pool images set pg_num 2000 ceph osd pool images set pgp_num 2000 ceph osd pool compute set size 3 ceph osd pool compute set min_size 3 ceph osd pool compute set pg_num 2000 ceph osd pool compute set pgp_num 2000 Here are the questions I am left with concerning these changes: 1. How long does it take for ceph to apply the changes and recalculate the pg’s? 2. When is it safe to do this type of operation? before any data is written to the pools or is doing this while pools are used acceptable? 3. Is it possible to scale down the number of pg’s ? Thank you for your input. Karol Kozubal ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] PG Scaling
Hi Everyone, I am deploying an openstack deployment with Fuel 4.1 and have a 20 node ceph deployment of c6220’s with 3 osd’s and 1 journaling disk per node. When first deployed each storage pool is configured with the correct size and min_size attributes however fuel doesn’t seem to apply the correct number of pg’s to the pools based on the number of osd’s that we actually have. I make the adjustments using the following (20 nodes * 3 OSDs)*100 / 3 replicas = 2000 ceph osd pool volumes set size 3 ceph osd pool volumes set min_size 3 ceph osd pool volumes set pg_num 2000 ceph osd pool volumes set pgp_num 2000 ceph osd pool images set size 3 ceph osd pool images set min_size 3 ceph osd pool images set pg_num 2000 ceph osd pool images set pgp_num 2000 ceph osd pool compute set size 3 ceph osd pool compute set min_size 3 ceph osd pool compute set pg_num 2000 ceph osd pool compute set pgp_num 2000 Here are the questions I am left with concerning these changes: 1. How long does it take for ceph to apply the changes and recalculate the pg’s? 2. When is it safe to do this type of operation? before any data is written to the pools or is doing this while pools are used acceptable? 3. Is it possible to scale down the number of pg’s ? Thank you for your input. Karol Kozubal ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] First Ceph Athens Meetup!
Hello everybody, we are happy to invite you to the first Ceph Athens meetup: http://www.meetup.com/Ceph-Athens on March 18th, 19:30, taking place on the 4th floor of the GRNET [1] HQ offices. We'll be happy to have Steve Starbuck of Inktank with us, who will introduce Ceph. Also, Vangelis Koukis from the Synnefo team will present how Ceph is being used to back GRNET’s large-scale, production, public cloud service called “~okeanos” [2]. So, if you want to learn more about Ceph, discuss or ask questions, feel free to join us! See you all there, Constantinos P.S.: Please, let us know if you're coming by joining the meetup on the above link. [1] http://www.grnet.gr/en [2] http://okeanos.grnet.gr ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] clock skew
2014-01-30 18:41 GMT+01:00 Eric Eastman eri...@aol.com: I have this problem on some of my Ceph clusters, and I think it is due to the older hardware the I am using does not have the best clocks. To fix the problem, I setup one server in my lab to be my local NTP time server, and then on each of my Ceph monitors, in the /etc/ntp.conf file, I put in a single server line that reads: server XX.XX.XX.XX iburst burst minpoll 4 maxpoll 5 I'm using a local NTP server, all Mons are synced with local NTP but ceph still detect a clock skew ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Wrong PG nums
Hi to all I have this in my conf: # grep 'pg num' /etc/ceph/ceph.conf osd pool default pg num = 5600 But: # ceph osd pool get data pg_num pg_num: 64 Is this normal ? Why just 64 pg was created ? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] neither POSIX nor REST
Hi, I saw radosgw-admin doesn't require radosgw daemon. I was wondering if by adding creation of buckets and manipulation of objects to radosgw-admin (or a library similar to it) ceph would have a great tool between the low and high levels of rados and radosgw, and simpler than cephfs. Of course those features, and object listing by prefix are not there for good reason that I don't know, but maybe it worths to be implemented and could make it easier for us (users) to innovate using ceph. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] clock skew
On Mar 12, 2014, at 10:44 AM, Gandalf Corvotempesta gandalf.corvotempe...@gmail.com wrote: 2014-01-30 18:41 GMT+01:00 Eric Eastman eri...@aol.com: I have this problem on some of my Ceph clusters, and I think it is due to the older hardware the I am using does not have the best clocks. To fix the problem, I setup one server in my lab to be my local NTP time server, and then on each of my Ceph monitors, in the /etc/ntp.conf file, I put in a single server line that reads: server XX.XX.XX.XX iburst burst minpoll 4 maxpoll 5 I'm using a local NTP server, all Mons are synced with local NTP but ceph still detect a clock skew Machine clocks aren't perfect, even with NTP. Ceph by default is very sensitive. I usually add this to my ceph.conf to prevent the warnings: [mon] mon clock drift allowed = .500 That is, allow the clocks to drift up to 1/2 second before saying anything. JN ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] radosgw where to put
Hello to everybody, does anbody know, if I need separate nodes to install radosgw (for example, to place into DMZ), or should I put radosgw daemons directly onto my storage nodes? Thank you in advance! Best wisches, Sergejs___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Put Ceph Cluster Behind a Pair of LB
From what I understand about Ceph architecture you would be causing a bottleneck for your ceph traffic. Ceph advantage is the potential concurrency of the traffic and the decentralization of the client facing interfaces increasing scale-out capabilities. Can you give a bit more details about your use case? On 2014-03-12, 1:32 PM, Larry Liu larryliu...@gmail.com wrote: Hi everyone, Anybody has a good practice on how to set up a ceph cluster behind a pair of load balancer? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] neither POSIX nor REST
On Wed, Mar 12, 2014 at 9:58 AM, Lorieri lori...@gmail.com wrote: Hi, I saw radosgw-admin doesn't require radosgw daemon. I was wondering if by adding creation of buckets and manipulation of objects to radosgw-admin (or a library similar to it) ceph would have a great tool between the low and high levels of rados and radosgw, and simpler than cephfs. Of course those features, and object listing by prefix are not there for good reason that I don't know, but maybe it worths to be implemented and could make it easier for us (users) to innovate using ceph. We discussed the possibility of creating a rgw library for object operations like that. Then it'd be easier to use it with radosgw-admin or to create other tools on top of it. This might be a good starter project as it's relatively easy to get something basic working. Yehuda ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Put Ceph Cluster Behind a Pair of LB
Anybody has a good practice on how to set up a ceph cluster behind a pair of load balancer? The only place you would want to put a load balancer in the context of a Ceph cluster would be north of RGW nodes. You can do L3 transparent load balancing or balance with a L7 proxy, ie Linux Virtual Server or HAProxy/Nginx. The other components of Ceph are horizontally scalable and because of the way Ceph's native protocols work you don't need load balancers doing L2/L3/L7 tricks to achieve HA. -- Kyle ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Put Ceph Cluster Behind a Pair of LB
You're right. Sorry didn't specify I was trying this for Radosgw. Even for this I'm seeing performance degrade once my clients start to hit the LB VIP. On Mar 12, 2014, at 10:57 AM, Kyle Bader kyle.ba...@gmail.com wrote: Anybody has a good practice on how to set up a ceph cluster behind a pair of load balancer? The only place you would want to put a load balancer in the context of a Ceph cluster would be north of RGW nodes. You can do L3 transparent load balancing or balance with a L7 proxy, ie Linux Virtual Server or HAProxy/Nginx. The other components of Ceph are horizontally scalable and because of the way Ceph's native protocols work you don't need load balancers doing L2/L3/L7 tricks to achieve HA. -- Kyle signature.asc Description: Message signed with OpenPGP using GPGMail ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Put Ceph Cluster Behind a Pair of LB
You're right. Sorry didn't specify I was trying this for Radosgw. Even for this I'm seeing performance degrade once my clients start to hit the LB VIP. Could you tell us more about your load balancer and configuration? -- Kyle ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Put Ceph Cluster Behind a Pair of LB
This is in my lab. Plain passthrough setup with automap enabled on the F5. s3 curl work fine as far as queries go. But file transfer rate degrades badly once I start file up/download. On Mar 12, 2014, at 11:12 AM, Kyle Bader kyle.ba...@gmail.com wrote: You're right. Sorry didn't specify I was trying this for Radosgw. Even for this I'm seeing performance degrade once my clients start to hit the LB VIP. Could you tell us more about your load balancer and configuration? -- Kyle signature.asc Description: Message signed with OpenPGP using GPGMail ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] if partition name changes, will ceph get corrupted?
We use /dev/disk/by-path for this reason, but we confirmed that is stable for our HBAs. Maybe /dev/disk/by-something is consistent with your controller. Cheers, Dan -- Dan van der Ster || Data Storage Services || CERN IT Department -- Original Message From: Sidharta Mukerjee smukerje...@gmail.com Sent: Wednesday, March 12, 2014 07:51 PM To: ceph-us...@ceph.com Subject: [ceph-users] if partition name changes, will ceph get corrupted? If a partition name such as /dev/sdd changes to /dev/sde and ceph was already mapped to the old /dev/sdd, how will CEPH react? For example, would it get corrupted, or notice a problem and fail remove that one OSD from cluster or somehow automatically re-adapt? FYI: I ask because we added a new hard disk (disk #3) in the middle of a 8-disk server and I'm not sure if the PERC RAID controller and/or the Centos 6.5 OS will rename the original parititions so that the partitions are names alphabetically or just add a new paritition (/dev/sdh) for this 3rd disk. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Put Ceph Cluster Behind a Pair of LB
This is in my lab. Plain passthrough setup with automap enabled on the F5. s3 curl work fine as far as queries go. But file transfer rate degrades badly once I start file up/download. Maybe the difference can be attributed to LAN client traffic with jumbo frames vs F5 using a smaller WAN MTU? -- Kyle ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] PG Scaling
Awesome thanks for the info. We have just began testing phase. I have 10Gig interfaces on both the cluster and public interfaces and using fast disks so I probably won’t feel much of a difference. Since this is just a test setup I have some freedom here but nice to know the consequences. Karol From: McNamara, Bradley bradley.mcnam...@seattle.govmailto:bradley.mcnam...@seattle.gov Date: Wednesday, March 12, 2014 at 7:01 PM To: Karol Kozubal karol.kozu...@elits.commailto:karol.kozu...@elits.com, ceph-users@lists.ceph.commailto:ceph-users@lists.ceph.com ceph-users@lists.ceph.commailto:ceph-users@lists.ceph.com Subject: RE: PG Scaling Most things will cause data movement… If you are going to have different failure zones within your crush map, I would edit your crush map and define those failure zones/buckets, first. This will cause data movement when you inject the new crush map into the cluster. This will immediately cause data movement. Once the data movement from the new crush map is done, then I would change the number of placement groups. This will immediately cause data movement, too. If you have a cluster network defined and in use, this shouldn’t materially affect the running cluster. Response times may be exaggerated, but the cluster will be completely functional. Brad From: Karol Kozubal [mailto:karol.kozu...@elits.com] Sent: Wednesday, March 12, 2014 1:52 PM To: McNamara, Bradley; ceph-users@lists.ceph.commailto:ceph-users@lists.ceph.com Subject: Re: PG Scaling Thank you for your response. The number of replicas is already set to 3. So if I simply increase the number of pg’s they will also start to move or is that simply triggered with size alterations? I suppose since this will generate movement in the cluster network it is ideal to do this operation while the cluster isnt as busy. Karol From: McNamara, Bradley bradley.mcnam...@seattle.govmailto:bradley.mcnam...@seattle.gov Date: Wednesday, March 12, 2014 at 1:54 PM To: Karol Kozubal karol.kozu...@elits.commailto:karol.kozu...@elits.com, ceph-users@lists.ceph.commailto:ceph-users@lists.ceph.com ceph-users@lists.ceph.commailto:ceph-users@lists.ceph.com Subject: RE: PG Scaling Round up your pg_num and pgp_num to the next power of 2, 2048. Ceph will start moving data as soon as you implement the new ‘size 3’, so I would increase the pg_num and pgp_num, first, then increase the size. It will start creating the new PG’s immediately. You can see all this going on using ‘ceph –w’. Once the data is finished moving, you may need to run ‘ceph osd crush tunables optimal’. This should take care of any unclean PG’s that may be hanging around. It is NOT possible to decrease the PG’s. One would need to delete the pool and recreate it. Brad From:ceph-users-boun...@lists.ceph.commailto:ceph-users-boun...@lists.ceph.com [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Karol Kozubal Sent: Wednesday, March 12, 2014 9:08 AM To: ceph-users@lists.ceph.commailto:ceph-users@lists.ceph.com Subject: Re: [ceph-users] PG Scaling Correction: Sorry min_size is at 1 everywhere. Thank you. Karol Kozubal From: Karol Kozubal karol.kozu...@elits.commailto:karol.kozu...@elits.com Date: Wednesday, March 12, 2014 at 12:06 PM To: ceph-users@lists.ceph.commailto:ceph-users@lists.ceph.com ceph-users@lists.ceph.commailto:ceph-users@lists.ceph.com Subject: PG Scaling Hi Everyone, I am deploying an openstack deployment with Fuel 4.1 and have a 20 node ceph deployment of c6220’s with 3 osd’s and 1 journaling disk per node. When first deployed each storage pool is configured with the correct size and min_size attributes however fuel doesn’t seem to apply the correct number of pg’s to the pools based on the number of osd’s that we actually have. I make the adjustments using the following (20 nodes * 3 OSDs)*100 / 3 replicas = 2000 ceph osd pool volumes set size 3 ceph osd pool volumes set min_size 3 ceph osd pool volumes set pg_num 2000 ceph osd pool volumes set pgp_num 2000 ceph osd pool images set size 3 ceph osd pool images set min_size 3 ceph osd pool images set pg_num 2000 ceph osd pool images set pgp_num 2000 ceph osd pool compute set size 3 ceph osd pool compute set min_size 3 ceph osd pool compute set pg_num 2000 ceph osd pool compute set pgp_num 2000 Here are the questions I am left with concerning these changes: 1. How long does it take for ceph to apply the changes and recalculate the pg’s? 2. When is it safe to do this type of operation? before any data is written to the pools or is doing this while pools are used acceptable? 3. Is it possible to scale down the number of pg’s ? Thank you for your input. Karol Kozubal ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] no user info saved after user creation / can't create buckets
And the debug log because that last log was obviously not helpful... 2014-03-12 23:57:49.497780 7ff97e7dd700 1 == starting new request req=0x23bc650 = 2014-03-12 23:57:49.498198 7ff97e7dd700 2 req 1:0.000419::PUT /test::initializing 2014-03-12 23:57:49.498233 7ff97e7dd700 10 host=s3.amazonaws.comrgw_dns_name=us-west-1.domain 2014-03-12 23:57:49.498366 7ff97e7dd700 10 s-object=NULL s-bucket=test 2014-03-12 23:57:49.498437 7ff97e7dd700 2 req 1:0.000659:s3:PUT /test::getting op 2014-03-12 23:57:49.498448 7ff97e7dd700 2 req 1:0.000670:s3:PUT /test:create_bucket:authorizing 2014-03-12 23:57:49.498508 7ff97e7dd700 10 cache get: name=.us-west-1.users+BLAHBLAHBLAH : miss 2014-03-12 23:57:49.500852 7ff97e7dd700 10 cache put: name=.us-west-1.users+BLAHBLAHBLAH 2014-03-12 23:57:49.500865 7ff97e7dd700 10 adding .us-west-1.users+BLAHBLAHBLAH to cache LRU end 2014-03-12 23:57:49.500886 7ff97e7dd700 10 moving .us-west-1.users+BLAHBLAHBLAH to cache LRU end 2014-03-12 23:57:49.500889 7ff97e7dd700 10 cache get: name=.us-west-1.users+BLAHBLAHBLAH : type miss (requested=1, cached=6) 2014-03-12 23:57:49.500907 7ff97e7dd700 10 moving .us-west-1.users+BLAHBLAHBLAH to cache LRU end 2014-03-12 23:57:49.500910 7ff97e7dd700 10 cache get: name=.us-west-1.users+BLAHBLAHBLAH : hit 2014-03-12 23:57:49.502663 7ff97e7dd700 10 cache put: name=.us-west-1.users+BLAHBLAHBLAH 2014-03-12 23:57:49.502667 7ff97e7dd700 10 moving .us-west-1.users+BLAHBLAHBLAH to cache LRU end 2014-03-12 23:57:49.502700 7ff97e7dd700 10 cache get: name=.us-west-1.users.uid+test : miss 2014-03-12 23:57:49.505128 7ff97e7dd700 10 cache put: name=.us-west-1.users.uid+test 2014-03-12 23:57:49.505138 7ff97e7dd700 10 adding .us-west-1.users.uid+test to cache LRU end 2014-03-12 23:57:49.505157 7ff97e7dd700 10 moving .us-west-1.users.uid+test to cache LRU end 2014-03-12 23:57:49.505160 7ff97e7dd700 10 cache get: name=.us-west-1.users.uid+test : type miss (requested=1, cached=6) 2014-03-12 23:57:49.505176 7ff97e7dd700 10 moving .us-west-1.users.uid+test to cache LRU end 2014-03-12 23:57:49.505178 7ff97e7dd700 10 cache get: name=.us-west-1.users.uid+test : hit 2014-03-12 23:57:49.507401 7ff97e7dd700 10 cache put: name=.us-west-1.users.uid+test 2014-03-12 23:57:49.507406 7ff97e7dd700 10 moving .us-west-1.users.uid+test to cache LRU end 2014-03-12 23:57:49.507521 7ff97e7dd700 10 get_canon_resource(): dest=/test 2014-03-12 23:57:49.507529 7ff97e7dd700 10 auth_hdr: PUT binary/octet-stream Wed, 12 Mar 2014 23:57:51 GMT /test 2014-03-12 23:57:49.507674 7ff97e7dd700 2 req 1:0.009895:s3:PUT /test:create_bucket:reading permissions 2014-03-12 23:57:49.507682 7ff97e7dd700 2 req 1:0.009904:s3:PUT /test:create_bucket:verifying op mask 2014-03-12 23:57:49.507695 7ff97e7dd700 2 req 1:0.009917:s3:PUT /test:create_bucket:verifying op permissions 2014-03-12 23:57:49.509604 7ff97e7dd700 2 req 1:0.011826:s3:PUT /test:create_bucket:verifying op params 2014-03-12 23:57:49.509615 7ff97e7dd700 2 req 1:0.011836:s3:PUT /test:create_bucket:executing 2014-03-12 23:57:49.509694 7ff97e7dd700 10 cache get: name=.us-west-1.domain.rgw+test : miss 2014-03-12 23:57:49.512229 7ff97e7dd700 10 cache put: name=.us-west-1.domain.rgw+test 2014-03-12 23:57:49.512259 7ff97e7dd700 10 adding .us-west-1.domain.rgw+test to cache LRU end 2014-03-12 23:57:49.512333 7ff97e7dd700 10 cache get: name=.us-west-1.domain.rgw+.pools.avail : miss 2014-03-12 23:57:49.518216 7ff97e7dd700 10 cache put: name=.us-west-1.domain.rgw+.pools.avail 2014-03-12 23:57:49.518228 7ff97e7dd700 10 adding .us-west-1.domain.rgw+.pools.avail to cache LRU end 2014-03-12 23:57:49.518248 7ff97e7dd700 10 moving .us-west-1.domain.rgw+.pools.avail to cache LRU end 2014-03-12 23:57:49.518251 7ff97e7dd700 10 cache get: name=.us-west-1.domain.rgw+.pools.avail : type miss (requested=1, cached=6) 2014-03-12 23:57:49.518270 7ff97e7dd700 10 moving .us-west-1.domain.rgw+.pools.avail to cache LRU end 2014-03-12 23:57:49.518272 7ff97e7dd700 10 cache get: name=.us-west-1.domain.rgw+.pools.avail : hit 2014-03-12 23:57:49.520295 7ff97e7dd700 10 cache put: name=.us-west-1.domain.rgw+.pools.avail 2014-03-12 23:57:49.520348 7ff97e7dd700 10 moving .us-west-1.domain.rgw+.pools.avail to cache LRU end 2014-03-12 23:57:49.522672 7ff97e7dd700 2 req 1:0.024893:s3:PUT /test:create_bucket:http status=403 2014-03-12 23:57:49.523204 7ff97e7dd700 1 == req done req=0x23bc650 http_status=403 == On Wed, Mar 12, 2014 at 7:36 PM, Greg Poirier greg.poir...@opower.comwrote: The saga continues... So, after fiddling with haproxy a bit, I managed to make sure that my requests were hitting the RADOS Gateway. NOW, I get a 403 from my ruby script: 2014-03-12 23:34:08.289670 7fda9bfbf700 1 == starting new request req=0x215a780 = 2014-03-12 23:34:08.305105 7fda9bfbf700 1 == req done req=0x215a780 http_status=403 == The aws-s3 gem forces the Host header to be set to s3.amazonaws.com -- and I am wondering if this could potentially cause a
Re: [ceph-users] no user info saved after user creation / can't create buckets
Increasing the logging further, and I notice the following: 2014-03-13 00:27:28.617100 7f6036ffd700 20 rgw_create_bucket returned ret=-1 bucket=test(@.rgw.buckets[us-west-1.15849318.1]) But hope that .rgw.buckets doesn't have to exist... and that @.rgw.buckets is perhaps telling of something? I did notice that .us-west-1.rgw.buckets and .us-west-1.rgw.buckets.index weren't created. I created those, restarted radosgw, and still 403 errors. On Wed, Mar 12, 2014 at 8:00 PM, Greg Poirier greg.poir...@opower.comwrote: And the debug log because that last log was obviously not helpful... 2014-03-12 23:57:49.497780 7ff97e7dd700 1 == starting new request req=0x23bc650 = 2014-03-12 23:57:49.498198 7ff97e7dd700 2 req 1:0.000419::PUT /test::initializing 2014-03-12 23:57:49.498233 7ff97e7dd700 10 host=s3.amazonaws.comrgw_dns_name=us-west-1.domain 2014-03-12 23:57:49.498366 7ff97e7dd700 10 s-object=NULL s-bucket=test 2014-03-12 23:57:49.498437 7ff97e7dd700 2 req 1:0.000659:s3:PUT /test::getting op 2014-03-12 23:57:49.498448 7ff97e7dd700 2 req 1:0.000670:s3:PUT /test:create_bucket:authorizing 2014-03-12 23:57:49.498508 7ff97e7dd700 10 cache get: name=.us-west-1.users+BLAHBLAHBLAH : miss 2014-03-12 23:57:49.500852 7ff97e7dd700 10 cache put: name=.us-west-1.users+BLAHBLAHBLAH 2014-03-12 23:57:49.500865 7ff97e7dd700 10 adding .us-west-1.users+BLAHBLAHBLAH to cache LRU end 2014-03-12 23:57:49.500886 7ff97e7dd700 10 moving .us-west-1.users+BLAHBLAHBLAH to cache LRU end 2014-03-12 23:57:49.500889 7ff97e7dd700 10 cache get: name=.us-west-1.users+BLAHBLAHBLAH : type miss (requested=1, cached=6) 2014-03-12 23:57:49.500907 7ff97e7dd700 10 moving .us-west-1.users+BLAHBLAHBLAH to cache LRU end 2014-03-12 23:57:49.500910 7ff97e7dd700 10 cache get: name=.us-west-1.users+BLAHBLAHBLAH : hit 2014-03-12 23:57:49.502663 7ff97e7dd700 10 cache put: name=.us-west-1.users+BLAHBLAHBLAH 2014-03-12 23:57:49.502667 7ff97e7dd700 10 moving .us-west-1.users+BLAHBLAHBLAH to cache LRU end 2014-03-12 23:57:49.502700 7ff97e7dd700 10 cache get: name=.us-west-1.users.uid+test : miss 2014-03-12 23:57:49.505128 7ff97e7dd700 10 cache put: name=.us-west-1.users.uid+test 2014-03-12 23:57:49.505138 7ff97e7dd700 10 adding .us-west-1.users.uid+test to cache LRU end 2014-03-12 23:57:49.505157 7ff97e7dd700 10 moving .us-west-1.users.uid+test to cache LRU end 2014-03-12 23:57:49.505160 7ff97e7dd700 10 cache get: name=.us-west-1.users.uid+test : type miss (requested=1, cached=6) 2014-03-12 23:57:49.505176 7ff97e7dd700 10 moving .us-west-1.users.uid+test to cache LRU end 2014-03-12 23:57:49.505178 7ff97e7dd700 10 cache get: name=.us-west-1.users.uid+test : hit 2014-03-12 23:57:49.507401 7ff97e7dd700 10 cache put: name=.us-west-1.users.uid+test 2014-03-12 23:57:49.507406 7ff97e7dd700 10 moving .us-west-1.users.uid+test to cache LRU end 2014-03-12 23:57:49.507521 7ff97e7dd700 10 get_canon_resource(): dest=/test 2014-03-12 23:57:49.507529 7ff97e7dd700 10 auth_hdr: PUT binary/octet-stream Wed, 12 Mar 2014 23:57:51 GMT /test 2014-03-12 23:57:49.507674 7ff97e7dd700 2 req 1:0.009895:s3:PUT /test:create_bucket:reading permissions 2014-03-12 23:57:49.507682 7ff97e7dd700 2 req 1:0.009904:s3:PUT /test:create_bucket:verifying op mask 2014-03-12 23:57:49.507695 7ff97e7dd700 2 req 1:0.009917:s3:PUT /test:create_bucket:verifying op permissions 2014-03-12 23:57:49.509604 7ff97e7dd700 2 req 1:0.011826:s3:PUT /test:create_bucket:verifying op params 2014-03-12 23:57:49.509615 7ff97e7dd700 2 req 1:0.011836:s3:PUT /test:create_bucket:executing 2014-03-12 23:57:49.509694 7ff97e7dd700 10 cache get: name=.us-west-1.domain.rgw+test : miss 2014-03-12 23:57:49.512229 7ff97e7dd700 10 cache put: name=.us-west-1.domain.rgw+test 2014-03-12 23:57:49.512259 7ff97e7dd700 10 adding .us-west-1.domain.rgw+test to cache LRU end 2014-03-12 23:57:49.512333 7ff97e7dd700 10 cache get: name=.us-west-1.domain.rgw+.pools.avail : miss 2014-03-12 23:57:49.518216 7ff97e7dd700 10 cache put: name=.us-west-1.domain.rgw+.pools.avail 2014-03-12 23:57:49.518228 7ff97e7dd700 10 adding .us-west-1.domain.rgw+.pools.avail to cache LRU end 2014-03-12 23:57:49.518248 7ff97e7dd700 10 moving .us-west-1.domain.rgw+.pools.avail to cache LRU end 2014-03-12 23:57:49.518251 7ff97e7dd700 10 cache get: name=.us-west-1.domain.rgw+.pools.avail : type miss (requested=1, cached=6) 2014-03-12 23:57:49.518270 7ff97e7dd700 10 moving .us-west-1.domain.rgw+.pools.avail to cache LRU end 2014-03-12 23:57:49.518272 7ff97e7dd700 10 cache get: name=.us-west-1.domain.rgw+.pools.avail : hit 2014-03-12 23:57:49.520295 7ff97e7dd700 10 cache put: name=.us-west-1.domain.rgw+.pools.avail 2014-03-12 23:57:49.520348 7ff97e7dd700 10 moving .us-west-1.domain.rgw+.pools.avail to cache LRU end 2014-03-12 23:57:49.522672 7ff97e7dd700 2 req 1:0.024893:s3:PUT /test:create_bucket:http status=403 2014-03-12 23:57:49.523204 7ff97e7dd700 1 == req
Re: [ceph-users] no user info saved after user creation / can't create buckets
And, I figured out the issue. The utility I was using to create pools, zones, and regions automatically failed to do two things: - create rgw.buckets and rgw.buckets.index for each zone - setup placement pools for each zone I did both of those, and now everything is working. Thanks, me, for the commitment to figuring this poo out. On Wed, Mar 12, 2014 at 8:31 PM, Greg Poirier greg.poir...@opower.comwrote: Increasing the logging further, and I notice the following: 2014-03-13 00:27:28.617100 7f6036ffd700 20 rgw_create_bucket returned ret=-1 bucket=test(@.rgw.buckets[us-west-1.15849318.1]) But hope that .rgw.buckets doesn't have to exist... and that @.rgw.buckets is perhaps telling of something? I did notice that .us-west-1.rgw.buckets and .us-west-1.rgw.buckets.index weren't created. I created those, restarted radosgw, and still 403 errors. On Wed, Mar 12, 2014 at 8:00 PM, Greg Poirier greg.poir...@opower.comwrote: And the debug log because that last log was obviously not helpful... 2014-03-12 23:57:49.497780 7ff97e7dd700 1 == starting new request req=0x23bc650 = 2014-03-12 23:57:49.498198 7ff97e7dd700 2 req 1:0.000419::PUT /test::initializing 2014-03-12 23:57:49.498233 7ff97e7dd700 10 host=s3.amazonaws.comrgw_dns_name=us-west-1.domain 2014-03-12 23:57:49.498366 7ff97e7dd700 10 s-object=NULL s-bucket=test 2014-03-12 23:57:49.498437 7ff97e7dd700 2 req 1:0.000659:s3:PUT /test::getting op 2014-03-12 23:57:49.498448 7ff97e7dd700 2 req 1:0.000670:s3:PUT /test:create_bucket:authorizing 2014-03-12 23:57:49.498508 7ff97e7dd700 10 cache get: name=.us-west-1.users+BLAHBLAHBLAH : miss 2014-03-12 23:57:49.500852 7ff97e7dd700 10 cache put: name=.us-west-1.users+BLAHBLAHBLAH 2014-03-12 23:57:49.500865 7ff97e7dd700 10 adding .us-west-1.users+BLAHBLAHBLAH to cache LRU end 2014-03-12 23:57:49.500886 7ff97e7dd700 10 moving .us-west-1.users+BLAHBLAHBLAH to cache LRU end 2014-03-12 23:57:49.500889 7ff97e7dd700 10 cache get: name=.us-west-1.users+BLAHBLAHBLAH : type miss (requested=1, cached=6) 2014-03-12 23:57:49.500907 7ff97e7dd700 10 moving .us-west-1.users+BLAHBLAHBLAH to cache LRU end 2014-03-12 23:57:49.500910 7ff97e7dd700 10 cache get: name=.us-west-1.users+BLAHBLAHBLAH : hit 2014-03-12 23:57:49.502663 7ff97e7dd700 10 cache put: name=.us-west-1.users+BLAHBLAHBLAH 2014-03-12 23:57:49.502667 7ff97e7dd700 10 moving .us-west-1.users+BLAHBLAHBLAH to cache LRU end 2014-03-12 23:57:49.502700 7ff97e7dd700 10 cache get: name=.us-west-1.users.uid+test : miss 2014-03-12 23:57:49.505128 7ff97e7dd700 10 cache put: name=.us-west-1.users.uid+test 2014-03-12 23:57:49.505138 7ff97e7dd700 10 adding .us-west-1.users.uid+test to cache LRU end 2014-03-12 23:57:49.505157 7ff97e7dd700 10 moving .us-west-1.users.uid+test to cache LRU end 2014-03-12 23:57:49.505160 7ff97e7dd700 10 cache get: name=.us-west-1.users.uid+test : type miss (requested=1, cached=6) 2014-03-12 23:57:49.505176 7ff97e7dd700 10 moving .us-west-1.users.uid+test to cache LRU end 2014-03-12 23:57:49.505178 7ff97e7dd700 10 cache get: name=.us-west-1.users.uid+test : hit 2014-03-12 23:57:49.507401 7ff97e7dd700 10 cache put: name=.us-west-1.users.uid+test 2014-03-12 23:57:49.507406 7ff97e7dd700 10 moving .us-west-1.users.uid+test to cache LRU end 2014-03-12 23:57:49.507521 7ff97e7dd700 10 get_canon_resource(): dest=/test 2014-03-12 23:57:49.507529 7ff97e7dd700 10 auth_hdr: PUT binary/octet-stream Wed, 12 Mar 2014 23:57:51 GMT /test 2014-03-12 23:57:49.507674 7ff97e7dd700 2 req 1:0.009895:s3:PUT /test:create_bucket:reading permissions 2014-03-12 23:57:49.507682 7ff97e7dd700 2 req 1:0.009904:s3:PUT /test:create_bucket:verifying op mask 2014-03-12 23:57:49.507695 7ff97e7dd700 2 req 1:0.009917:s3:PUT /test:create_bucket:verifying op permissions 2014-03-12 23:57:49.509604 7ff97e7dd700 2 req 1:0.011826:s3:PUT /test:create_bucket:verifying op params 2014-03-12 23:57:49.509615 7ff97e7dd700 2 req 1:0.011836:s3:PUT /test:create_bucket:executing 2014-03-12 23:57:49.509694 7ff97e7dd700 10 cache get: name=.us-west-1.domain.rgw+test : miss 2014-03-12 23:57:49.512229 7ff97e7dd700 10 cache put: name=.us-west-1.domain.rgw+test 2014-03-12 23:57:49.512259 7ff97e7dd700 10 adding .us-west-1.domain.rgw+test to cache LRU end 2014-03-12 23:57:49.512333 7ff97e7dd700 10 cache get: name=.us-west-1.domain.rgw+.pools.avail : miss 2014-03-12 23:57:49.518216 7ff97e7dd700 10 cache put: name=.us-west-1.domain.rgw+.pools.avail 2014-03-12 23:57:49.518228 7ff97e7dd700 10 adding .us-west-1.domain.rgw+.pools.avail to cache LRU end 2014-03-12 23:57:49.518248 7ff97e7dd700 10 moving .us-west-1.domain.rgw+.pools.avail to cache LRU end 2014-03-12 23:57:49.518251 7ff97e7dd700 10 cache get: name=.us-west-1.domain.rgw+.pools.avail : type miss (requested=1, cached=6) 2014-03-12 23:57:49.518270 7ff97e7dd700 10 moving .us-west-1.domain.rgw+.pools.avail to cache LRU end 2014-03-12 23:57:49.518272
Re: [ceph-users] Access Denied errors
You need to try switching the calling format. Instead of accessing http://admin.XXX/user, try accessing http://XXX/admin/user. There's a good chance that the subdomain calling format does not work with the admin api. Yehuda On Wed, Mar 12, 2014 at 8:58 PM, Steve Carter scar...@liquidweb.com wrote: Any ideas on this? Anything I can do to further troubleshoot? -Steve From: Steve Carter scar...@liquidweb.com To: Yehuda Sadeh yeh...@inktank.com Cc: ceph-users@lists.ceph.com Sent: Tuesday, March 11, 2014 4:42:24 PM Subject: Re: [ceph-users] Access Denied errors Just to be complete, a TCP Dump: Starting tcpick 0.2.1 at 2014-03-11 21:11 UTC Timeout for connections is 600 tcpick: reading from test.pcap 1 SYN-SENT 10.255.247.241:39729 10.30.77.227:http 1 SYN-RECEIVED 10.255.247.241:39729 10.30.77.227:http 1 ESTABLISHED10.255.247.241:39729 10.30.77.227:http GET /user HTTP/1.1 TE: deflate,gzip;q=0.3 Keep-Alive: 300 Connection: Keep-Alive, TE Date: Mon, 10 Mar 2014 22:51:06 GMT Authorization: AWS 08V6K45V9KPVK7MIWWMG:tot0rXT4AeYohcRQ0iyGPnAQ+cg= Host: admin..liquidweb.com User-Agent: libwww-perl/5.805 display-name: Hello World uid: atc HTTP/1.1 403 Forbidden Date: Mon, 10 Mar 2014 22:50:36 GMT Server: Apache/2.2.22 (Ubuntu) Accept-Ranges: bytes Content-Length: 78 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: application/xml ?xml version=1.0 encoding=UTF-8?ErrorCodeAccessDenied/Code/Error 1 FIN-WAIT-1 10.255.247.241:39729 10.30.77.227:http 1 TIME-WAIT 10.255.247.241:39729 10.30.77.227:http 1 CLOSED 10.255.247.241:39729 10.30.77.227:http tcpick: done reading from test.pcap 10 packets captured 1 tcp sessions detected From: Steve Carter scar...@liquidweb.com To: Yehuda Sadeh yeh...@inktank.com Cc: ceph-users@lists.ceph.com Sent: Tuesday, March 11, 2014 4:35:12 PM Subject: Re: [ceph-users] Access Denied errors On Mar 10, 2014, at 8:30 PM, Yehuda Sadeh yeh...@inktank.com wrote: 2014-03-10 22:59:12.551012 7fec017fa700 10 auth_hdr: GET Mon, 10 Mar 2014 22:59:42 GMT /user This is related to the issue. I assume it was signed as /admin/user, but here we just use /user because that what's passed in the URI. Are you accessing the gateway through virtual dns bucket name (e.g., admin.your-domain.com)? Yehuda 2014-03-10 22:59:12.551103 7fec017fa700 15 calculated digest=R+4z9J6PyXugdHAYJDKJiLPKpWo= 2014-03-10 22:59:12.551113 7fec017fa700 15 auth_sign=OHAxWvf8U8t4CVWq0pKKwxZ2Xko= 2014-03-10 22:59:12.551114 7fec017fa700 15 compare=-3 2014-03-10 22:59:12.551118 7fec017fa700 10 failed to authorize request 2014-03-10 22:59:12.551295 7fec017fa700 2 req 1:0.020363:s3:GET /user:list_bucket:http status=403 2014-03-10 22:59:12.551496 7fec017fa700 1 == req done req=0x19497c0 http_status=403 == This what our request header looks like. 'admin' is the admin bucket. The request doesn't appear to be signed as /admin/user. I wonder if the ordering of our header fields are incorrect insofar as they don't match the canonical ordering expected by radosgw/S3 resulting in the digests not matching? Request: GET http://admin..liquidweb.com/user Date: Tue, 11 Mar 2014 22:52:20 GMT Authorization: AWS 08V6K45V9KPVK7MIWWMG:VPPhzMiF9bFywTxLbr1peLEwZK4= User-Agent: libwww-perl/5.805 display-name: Hello World uid: atc Format: json HTTP/1.1 *** /home/etank/lwlibs/perl/Amazon/S3.pm [298]: Response: HTTP/1.1 403 Forbidden Connection: Keep-Alive Date: Tue, 11 Mar 2014 22:51:47 GMT Accept-Ranges: bytes Server: Apache/2.2.22 (Ubuntu) Content-Length: 78 Content-Type: application/xml Client-Date: Tue, 11 Mar 2014 22:52:20 GMT Client-Peer: 10.30.77.227:80 Client-Response-Num: 1 Keep-Alive: timeout=5, max=100 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
