Re: [ceph-users] rbd on ubuntu 12.04 LTS
I tryed with:
ceph osd crush tunables default
ceph osd crush tunables argonaut
while the command runs without error, I still get the feature set mismatch
error whe I try to mount
do I have to restart some service?
Andi
> -Original Message-
> From: Gregory Farnum [mailto:g...@inktank.com]
> Sent: Donnerstag, 31. Oktober 2013 18:07
> To: Fuchs, Andreas (SwissTXT)
> Cc: Karan Singh; ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] rbd on ubuntu 12.04 LTS
>
> Well, looking at that doc page, with kernel 3.8 you can't use the newest set
> so you should set your profile to "argonaut" (the old ones). Or if you're
> feeling ambitious I believe you can turn on CRUSH_TUNABLES without
> CRUSH_TUNABLES2 by manually editing and injecting the CRUSH map (I
> haven't done this myself).
> Keep in mind that this is changing the data placement algorithm slightly, so
> it will result in data movement when you do this.
> -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com
>
> On Thu, Oct 31, 2013 at 9:35 AM, Fuchs, Andreas (SwissTXT)
> wrote:
> > Ok i halfway understand this.
> >
> >
> >
> > So I can either upgrade to a kernel version v3.9 or later
> >
> > Or
> >
> > Change the crushmap with ceph osd crush tunables {PROFILE}
> >
> > To a new profile.
> >
> >
> >
> > But to which profile do I have to change so that my ubuntu client is
> > supported?
> >
> >
> >
> > Andi
> >
> >
> >
> > From: Gregory Farnum [mailto:g...@inktank.com]
> > Sent: Donnerstag, 31. Oktober 2013 17:27
> >
> >
> > To: Fuchs, Andreas (SwissTXT)
> > Cc: Karan Singh; ceph-users@lists.ceph.com
> > Subject: Re: [ceph-users] rbd on ubuntu 12.04 LTS
> >
> >
> >
> > Yeah, depending in what version of Ceph you deployed that could be it
> > exactly. We were a little more aggressive than we should have been in
> > pushing them out. See:
> >
> > http://ceph.com/docs/master/rados/operations/crush-map/#tunables
> >
> > -Greg
> >
> > On Thursday, October 31, 2013, Fuchs, Andreas (SwissTXT) wrote:
> >
> > Hmm i have a pretty much default install with ceph-deploy, the crusmap
> > is untouched.
> > Btw I get similar errormessages when trying to mount cephfs
> >
> > mount -t ceph ceph01:6789:/ /mnt/backupCephfs -o
> > name=admin,secretfile=admin.secret
> >
> > tail syslog
> > Oct 31 17:14:27 ceph00 kernel: [103642.162813] libceph: mon0
> > 10.100.214.11:6789 feature set mismatch, my 4008a < server's 4004008a,
> > missing 4000 Oct 31 17:14:27 ceph00 kernel: [103642.162937]
> > libceph: mon0
> > 10.100.214.11:6789 socket error on read Oct 31 17:14:36 ceph00 kernel:
> > [103651.151655] libceph: mon0
> > 10.100.214.11:6789 feature set mismatch, my 4008a < server's 4004008a,
> > missing 4000 Oct 31 17:14:36 ceph00 kernel: [103651.151761]
> > libceph: mon0
> > 10.100.214.11:6789 socket error on read Oct 31 17:14:46 ceph00 kernel:
> > [103661.142229] libceph: mon0
> > 10.100.214.11:6789 feature set mismatch, my 4008a < server's 4004008a,
> > missing 4000 Oct 31 17:14:46 ceph00 kernel: [103661.142334]
> > libceph: mon0
> > 10.100.214.11:6789 socket error on read
> >
> >
> > The client is Ubuntu 12.04 LTS
> > The OSD nodes are CentOS 6.4 kernel 3.8.0-32-generic
> >
> > Might this be an issue?
> >
> >> -Original Message-
> >> From: Gregory Farnum [mailto:g...@inktank.com]
> >> Sent: Mittwoch, 30. Oktober 2013 23:57
> >> To: Fuchs, Andreas (SwissTXT)
> >> Cc: Karan Singh; ceph-users@lists.ceph.com
> >> Subject: Re: [ceph-users] rbd on ubuntu 12.04 LTS
> >>
> >> You've enabled some feature on your cluster which is not supported by
> >> that kernel client. It's probably the crush tunables (you can find
> >> info on them in the docs).
> >> -Greg
> >> Software Engineer #42 @ http://inktank.com | http://ceph.com
> >>
> >>
> >> On Wed, Oct 30, 2013 at 3:59 AM, Fuchs, Andreas (SwissTXT)
> >> wrote:
> >> > I think keyring is fine, as i can run other commands like:
> >> >
> >> >
> >> >
> >> > rbd ls --pool rbd --name client.admin
> >> >
> >> > archiveadmin
> >> >
> >> >
> >> >
> >> > and was able to create the image
> >> >
> >> >
> >> >
> >> > rbd info archiveadmin --pool rbd --name client.admin
> >> >
> >> > rbd image 'archiveadmin':
> >> >
> >> > size 4096 MB in 1024 objects
> >> >
> >> > order 22 (4096 KB objects)
> >> >
> >> > block_name_prefix: rb.0.2c68.2ae8944a
> >> >
> >> > format: 1
> >> >
> >> >
> >> >
> >> > From: Karan Singh [mailto:ksi...@csc.fi]
> >> > Sent: Mittwoch, 30. Oktober 2013 10:57
> >> > To: Fuchs, Andreas (SwissTXT)
> >> > Cc: ceph-users@lists.ceph.com
> >> > Subject: Re: [ceph-users] rbd on ubuntu 12.04 LTS
> >> >
> >> >
> >> >
> >> > Andi
> >> >
> >> >
> >> >
> >> > Dont know much about but checking on keyrings side will give you
> >> > more information. Socket read problem usually caused by keyrings.
> >> >
> >> >
> >> >
> >> > Regards
> >> >
> >> > Karan
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > From: "Andreas Fuchs (SwissTXT)"
>
Re: [ceph-users] "rbd map" says "bat option at rw"
Hi Greg, I apologize for the lack of details. To sum up, I check that my image exists: $ rbd ls img0 img1 Then I try to mount it: $ sudo rbd map img0 rbd: add failed: (22) Invalid argument When I try the exact same command from the box with version 0.61.9, it succeeds: $ rbd ls img0 img1 $ sudo rbd map img0 $ rbd showmapped id pool image snap device 0 rbd img0 -/dev/rbd0 I have tried changing the data pool, the image format, the image size. I checked that the image was not locked, and not mounted anywhere else. I checked that the rbd kernel module was properly loaded, and I even tried from another box running 0.71 but I got the same error. I would love to do more troubleshooting myself, but the "Invalid argument" error message does not give me much to start with. Any hint? Best regards, Nicolas Canceill Scalable Storage Systems SURFsara (Amsterdam, NL) On 11/01/2013 06:10 PM, Gregory Farnum wrote: I think this will be easier to help with if you provide the exact command you're running. :) -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com On Fri, Nov 1, 2013 at 3:07 AM, nicolasc wrote: Hi every one, I finally and happily managed to get my Ceph cluster (3 monitors among 8 nodes, each with 9 OSDs) running on version 0.71, but the "rbd map" command shows a weird behaviour. I can list pools, create images and snapshots, alleluia! However, mapping to a device with "rbd map" is not working. When I try this from one of my nodes, the kernel says: libceph: bad option at 'rw' Which "rbd" translates into: add failed: (22) Invalid argument Any idea of what that could indicate? I am using a basic config: no authentication, default crushmap (I just changed some weights), and basic network config (public net, cluster net). I have tried both image formats, different sizes and pools. Moreover, I have a client running rbd from Ceph version 0.61.9, and from there everything works fine with "rbd map" on the same image. Both nodes (Ceph 0.61.9 and 0.71) are running Linux kernel 3.2 for Debian. Hope you can provide some hints. Best regards, Nicolas Canceill Scalable Storage Systems SURFsara (Amsterdam, NL) ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] "rbd map" says "bat option at rw"
I'm pretty sure this is the 'rw' mount option bug that Josh just fixed. It affects kernels older than 3.7 or so and Ceoh newer than .70 or .71 (somewhere in there). Can you try installing the package for the 'next' branch from gitbuilder.ceph.com? If you are using ceph-deploy you can do ceph-deploy install --dev=next HOST Thanks! sage nicolasc wrote: >Hi Greg, > >I apologize for the lack of details. To sum up, I check that my image >exists: > >$ rbd ls >img0 >img1 > >Then I try to mount it: > >$ sudo rbd map img0 >rbd: add failed: (22) Invalid argument > >When I try the exact same command from the box with version 0.61.9, it >succeeds: > >$ rbd ls >img0 >img1 >$ sudo rbd map img0 >$ rbd showmapped >id pool image snap device >0 rbd img0 -/dev/rbd0 > >I have tried changing the data pool, the image format, the image size. >I >checked that the image was not locked, and not mounted anywhere else. I > >checked that the rbd kernel module was properly loaded, and I even >tried >from another box running 0.71 but I got the same error. > >I would love to do more troubleshooting myself, but the "Invalid >argument" error message does not give me much to start with. Any hint? > >Best regards, > >Nicolas Canceill >Scalable Storage Systems >SURFsara (Amsterdam, NL) > > > >On 11/01/2013 06:10 PM, Gregory Farnum wrote: >> I think this will be easier to help with if you provide the exact >> command you're running. :) >> -Greg >> Software Engineer #42 @ http://inktank.com | http://ceph.com >> >> >> On Fri, Nov 1, 2013 at 3:07 AM, nicolasc > wrote: >>> Hi every one, >>> >>> I finally and happily managed to get my Ceph cluster (3 monitors >among 8 >>> nodes, each with 9 OSDs) running on version 0.71, but the "rbd map" >command >>> shows a weird behaviour. >>> >>> I can list pools, create images and snapshots, alleluia! >>> However, mapping to a device with "rbd map" is not working. When I >try this >>> from one of my nodes, the kernel says: >>> libceph: bad option at 'rw' >>> Which "rbd" translates into: >>> add failed: (22) Invalid argument >>> >>> Any idea of what that could indicate? >>> >>> I am using a basic config: no authentication, default crushmap (I >just >>> changed some weights), and basic network config (public net, cluster >net). I >>> have tried both image formats, different sizes and pools. >>> >>> Moreover, I have a client running rbd from Ceph version 0.61.9, and >from >>> there everything works fine with "rbd map" on the same image. Both >nodes >>> (Ceph 0.61.9 and 0.71) are running Linux kernel 3.2 for Debian. >>> >>> Hope you can provide some hints. Best regards, >>> >>> Nicolas Canceill >>> Scalable Storage Systems >>> SURFsara (Amsterdam, NL) >>> >>> ___ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >___ >ceph-users mailing list >ceph-users@lists.ceph.com >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Ceph deployment using puppet
Hello Everyone Can someone guide me how i can start for " ceph deployment using puppet " , what all things i need to have for this . I have no prior idea of using puppet , hence need your help to getting started with it. Regards Karan Singh ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] "rbd map" says "bat option at rw"
Hi Sage, Just tried it, the behaviour disappears in version 0.72-rc1, so it seems you got it right. Thanks for the reply! I did not see any mention of that bug in the 0.70 or 0.71 release notes, though. Keep up the good work. Best regards, Nicolas Canceill Scalable Storage Systems SURFsara (Amsterdam, NL) On 11/04/2013 11:27 AM, Sage Weil wrote: I'm pretty sure this is the 'rw' mount option bug that Josh just fixed. It affects kernels older than 3.7 or so and Ceoh newer than .70 or .71 (somewhere in there). Can you try installing the package for the 'next' branch from gitbuilder.ceph.com? If you are using ceph-deploy you can do ceph-deploy install --dev=next HOST Thanks! sage nicolasc wrote: Hi Greg, I apologize for the lack of details. To sum up, I check that my image exists: $ rbd ls img0 img1 Then I try to mount it: $ sudo rbd map img0 rbd: add failed: (22) Invalid argument When I try the exact same command from the box with version 0.61.9, it succeeds: $ rbd ls img0 img1 $ sudo rbd map img0 $ rbd showmapped id pool image snap device 0 rbd img0 -/dev/rbd0 I have tried changing the data pool, the image format, the image size. I checked that the image was not locked, and not mounted anywhere else. I checked that the rbd kernel module was properly loaded, and I even tried from another box running 0.71 but I got the same error. I would love to do more troubleshooting myself, but the "Invalid argument" error message does not give me much to start with. Any hint? Best regards, Nicolas Canceill Scalable Storage Systems SURFsara (Amsterdam, NL) On 11/01/2013 06:10 PM, Gregory Farnum wrote: I think this will be easier to help with if you provide the exact command you're running. :) -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com On Fri, Nov 1, 2013 at 3:07 AM, nicolasc wrote: Hi every one, I finally and happily managed to get my Ceph cluster (3 monitors among 8 nodes, each with 9 OSDs) running on version 0.71, but the "rbd map" command shows a weird behaviour. I can list pools, create images and snapshots, alleluia! However, mapping to a device with "rbd map" is not working. When I try this from one of my nodes, the kernel says: libceph: bad option at 'rw' Which "rbd" translates into: add failed: (22) Invalid argument Any idea of what that could indicate? I am using a basic config: no authentication, default crushmap (I just changed some weights), and basic network config (public net, cluster net). I have tried both image formats, different sizes and pools. Moreover, I have a client running rbd from Ceph version 0.61.9, and from there everything works fine with "rbd map" on the same image. Both nodes (Ceph 0.61.9 and 0.71) are running Linux kernel 3.2 for Debian. Hope you can provide some hints. Best regards, Nicolas Canceill Scalable Storage Systems SURFsara (Amsterdam, NL) ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] rgw bucket creation fails
Hi, using ceph 0.67.4 I followed http://ceph.com/docs/master/radosgw/. I can connect using s3cmd (test configuration succeeds), so the user credentials and everything else seems to be running as it should. But when doing a "s3cmd mb s3://test" the radosgw returns a "405 Method Not Allowed" (connection captured with wireshark). The radosgw logs don't contain anythin related to this. What am I doing wrong? Corin ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph deployment using puppet
Hi, Unless you're force to use puppet for some reason, I suggest you give ceph-deploy a try: http://ceph.com/docs/master/start/quick-ceph-deploy/ Cheers On 04/11/2013 19:00, Karan Singh wrote: > Hello Everyone > > Can someone guide me how i can start for " ceph deployment using puppet " , > what all things i need to have for this . > > I have no prior idea of using puppet , hence need your help to getting > started with it. > > > Regards > Karan Singh > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Loïc Dachary, Artisan Logiciel Libre signature.asc Description: OpenPGP digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Very frustrated with Ceph!
Are you saying despite the osdir error message (I am pasting again below from my posting yesterday) the OSDs are successfully prepared? [ceph_deploy.osd][ERROR ] OSError: [Errno 2] No such file or directory [ceph_deploy][ERROR ] GenericError: Failed to create 2 OSDs Thanks! Narendra -Original Message- From: Alfredo Deza [mailto:alfredo.d...@inktank.com] Sent: Saturday, November 02, 2013 12:03 PM To: Sage Weil Cc: Trivedi, Narendra; ceph-users@lists.ceph.com Subject: Re: [ceph-users] Very frustrated with Ceph! On Fri, Nov 1, 2013 at 11:12 PM, Sage Weil wrote: > On Sat, 2 Nov 2013, Trivedi, Narendra wrote: >> >> Hi Sage, >> >> I believe I issued a "ceph-deploy install..." from the admin node as >> per the documentation and that was almost ok as per the output of the >> command below except sometimes there?s an error followed by an ?OK? >> message (see the highlighted item in the red below). I eventually ran >> into some permission issues but seems things went okay: Maybe what can be confusing here is that ceph-deploy interprets stderr as ERROR logging level. Unfortunately, some tools will output normal informative data to stderr when they are clearly not errors. stdout, on the other hand, is interpreted by ceph-deploy as DEBUG level, so you will see logging at that level too. There is no way for ceph-deploy to tell if you are actually seeing errors because the tool is in fact sending error messages or because it decided to use stderr to send information that should go to stdout. > > Hmm, the below output makes it look like it was successfully installed > on > node1 node2 and node3. Can you confirm that /etc/ceph exists on all > three of those hosts? > > Oh, looking back at your original message, it looks like you are > trying to create OSDs on /tmp/osd*. I would create directories like > /ceph/osdo, /ceph/osd1, or similar. I believe you need to create the > directories beforehand, too. (In a normal deployment, you are either > feeding ceph raw disks (/dev/XXX) or an existing mount point on a > dedicated disk you already configured and mounted.) > > sage > > > > >> >> >> [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ceph-deploy install >> ceph-node1-mon-centos-6-4 ceph-node2-osd0-centos-6-4 >> ceph-node3-osd1-centos-6-4 >> >> [ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy install >> ceph-node1-mon-centos-6-4 ceph-node2-osd0-centos-6-4 >> ceph-node3-osd1-centos-6-4 >> >> [ceph_deploy.install][DEBUG ] Installing stable version dumpling on >> cluster ceph hosts ceph-node1-mon-centos-6-4 >> ceph-node2-osd0-centos-6-4 >> ceph-node3-osd1-centos-6-4 >> >> [ceph_deploy.install][DEBUG ] Detecting platform for host >> ceph-node1-mon-centos-6-4 ... >> >> [ceph-node1-mon-centos-6-4][DEBUG ] connected to host: >> ceph-node1-mon-centos-6-4 >> >> [ceph-node1-mon-centos-6-4][DEBUG ] detect platform information from >> remote host >> >> [ceph-node1-mon-centos-6-4][DEBUG ] detect machine type >> >> [ceph_deploy.install][INFO ] Distro info: CentOS 6.4 Final >> >> [ceph-node1-mon-centos-6-4][INFO ] installing ceph on >> ceph-node1-mon-centos-6-4 >> >> [ceph-node1-mon-centos-6-4][INFO ] adding EPEL repository >> >> [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo wget >> http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch >> .rpm >> >> [ceph-node1-mon-centos-6-4][ERROR ] --2013-11-01 19:51:20-- >> http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch >> .rpm >> >> [ceph-node1-mon-centos-6-4][ERROR ] Connecting to 10.12.132.208:8080... >> connected. >> >> [ceph-node1-mon-centos-6-4][ERROR ] Proxy request sent, awaiting response... >> 200 OK >> >> [ceph-node1-mon-centos-6-4][ERROR ] Length: 14540 (14K) >> [application/x-rpm] >> >> [ceph-node1-mon-centos-6-4][ERROR ] Saving to: >> `epel-release-6-8.noarch.rpm.2' >> >> [ceph-node1-mon-centos-6-4][ERROR ] >> >> [ceph-node1-mon-centos-6-4][ERROR ] 0K .. >> 100% 4.79M=0.003s >> >> [ceph-node1-mon-centos-6-4][ERROR ] >> >> [ceph-node1-mon-centos-6-4][ERROR ] Last-modified header invalid -- >> time-stamp ignored. >> >> [ceph-node1-mon-centos-6-4][ERROR ] 2013-11-01 19:52:20 (4.79 MB/s) - >> `epel-release-6-8.noarch.rpm.2' saved [14540/14540] >> >> [ceph-node1-mon-centos-6-4][ERROR ] >> >> [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo rpm -Uvh >> --replacepkgs epel-release-6*.rpm >> >> [ceph-node1-mon-centos-6-4][DEBUG ] Preparing... >> ## >> >> [ceph-node1-mon-centos-6-4][DEBUG ] epel-release >> ## >> >> [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo rpm >> --import >> https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc >> >> [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo rpm -Uvh >> --replacepkgs >> http://ceph.com/rpm-dumpling/el6/noarch/ceph-release-1-0.el6.noarch.r >> pm >
Re: [ceph-users] Ceph deployment using puppet
Hello Loic Thanks for your reply , Ceph-deploy works good to me. My next objective is to deploy ceph using puppet. Can you guide me now i can proceed. Regards karan - Original Message - From: "Loic Dachary" To: ceph-users@lists.ceph.com Sent: Monday, 4 November, 2013 4:45:06 PM Subject: Re: [ceph-users] Ceph deployment using puppet Hi, Unless you're force to use puppet for some reason, I suggest you give ceph-deploy a try: http://ceph.com/docs/master/start/quick-ceph-deploy/ Cheers On 04/11/2013 19:00, Karan Singh wrote: > Hello Everyone > > Can someone guide me how i can start for " ceph deployment using puppet " , > what all things i need to have for this . > > I have no prior idea of using puppet , hence need your help to getting > started with it. > > > Regards > Karan Singh > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Loïc Dachary, Artisan Logiciel Libre ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Very frustrated with Ceph!
Hello,
We experienced the same error as reported by Navendra, although we're
running Ubuntu Server 12.04.
We managed to work around the error (by trial and error). Below are the
steps we performed, perhaps this can help you track down the error.
*Step 1 - This was the error*
openstack@monitor3:~/cluster1$ *ceph-deploy -v osd prepare
ceph1:sde:/dev/sdb*
[ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy -v osd
prepare ceph1:sde:/dev/sdb
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks
ceph1:/dev/sde:/dev/sdb
[ceph1][DEBUG ] connected to host: ceph1
[ceph1][DEBUG ] detect platform information from remote host
[ceph1][DEBUG ] detect machine type
[ceph_deploy.osd][INFO ] Distro info: Ubuntu 12.04 precise
[ceph_deploy.osd][DEBUG ] Deploying osd to ceph1
[ceph1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph1][WARNIN] osd keyring does not exist yet, creating one
[ceph_deploy.osd][ERROR ] OSError: [Errno 2] No such file or directory
[ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs
*Step 2 - We then ran the following commands:*
Deploy-node : ceph-deploy uninstall ceph1
Ceph1-node : sudo rm –fr /etc/ceph/*
Deploy-node : ceph-deploy gatherkeys ceph1
Deploy-node : ceph-deploy -v install ceph1
*Step 3 - Apparently the problem was solved:*
Deploy-node : *ceph-deploy -v osd prepare ceph1:sde:/dev/sdb*
openstack@monitor3:~/cluster1$ ceph-deploy -v osd prepare ceph1:sde:/dev/sdb
[ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy -v osd
prepare ceph1:sde:/dev/sdb
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks
ceph1:/dev/sde:/dev/sdb
[ceph1][DEBUG ] connected to host: ceph1
[ceph1][DEBUG ] detect platform information from remote host
[ceph1][DEBUG ] detect machine type
[ceph_deploy.osd][INFO ] Distro info: Ubuntu 12.04 precise
[ceph_deploy.osd][DEBUG ] Deploying osd to ceph1
[ceph1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph1][WARNIN] osd keyring does not exist yet, creating one
[ceph1][INFO ] Running command: sudo udevadm trigger
--subsystem-match=block --action=add
[ceph_deploy.osd][DEBUG ] Preparing host ceph1 disk /dev/sde journal
/dev/sdb activate False
[ceph1][INFO ] Running command: sudo ceph-disk-prepare --fs-type xfs
--cluster ceph -- /dev/sde /dev/sdb
[ceph1][ERROR ] WARNING:ceph-disk:OSD will not be hot-swappable if journal
is not the same device as the osd data
[ceph1][DEBUG ] Information: Moved requested sector from 34 to 2048 in
[ceph1][DEBUG ] order to align on 2048-sector boundaries.
[ceph1][DEBUG ] The operation has completed successfully.
[ceph1][DEBUG ] Information: Moved requested sector from 34 to 2048 in
[ceph1][DEBUG ] order to align on 2048-sector boundaries.
[ceph1][DEBUG ] The operation has completed successfully.
[ceph1][DEBUG ] meta-data=/dev/sde1 isize=2048 agcount=4,
agsize=61047597 blks
[ceph1][DEBUG ] = sectsz=512 attr=2,
projid32bit=0
[ceph1][DEBUG ] data = bsize=4096
blocks=244190385, imaxpct=25
[ceph1][DEBUG ] = sunit=0 swidth=0 blks
[ceph1][DEBUG ] naming =version 2 bsize=4096 ascii-ci=0
[ceph1][DEBUG ] log =internal log bsize=4096
blocks=119233, version=2
[ceph1][DEBUG ] = sectsz=512 sunit=0 blks,
lazy-count=1
[ceph1][DEBUG ] realtime =none extsz=4096 blocks=0,
rtextents=0
[ceph1][DEBUG ] The operation has completed successfully.
[ceph_deploy.osd][DEBUG ] Host ceph1 is now ready for osd use.
Thanks!
Mike
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Very frustrated with Ceph!
On Mon, Nov 4, 2013 at 9:55 AM, Trivedi, Narendra wrote: > Are you saying despite the osdir error message (I am pasting again below from > my posting yesterday) the OSDs are successfully prepared? > > [ceph_deploy.osd][ERROR ] OSError: [Errno 2] No such file or directory > [ceph_deploy][ERROR ] GenericError: Failed to create 2 OSDs > Not these, they do look like genuine errors to me. I meant, remote commands (those appear to be local as you see `ceph_deploy` as the host). For example `wget` is a known offender (from your output): [ceph-node1-mon-centos-6-4][ERROR ] Proxy request sent, awaiting response... 200 OK > Thanks! > Narendra > -Original Message- > From: Alfredo Deza [mailto:alfredo.d...@inktank.com] > Sent: Saturday, November 02, 2013 12:03 PM > To: Sage Weil > Cc: Trivedi, Narendra; ceph-users@lists.ceph.com > Subject: Re: [ceph-users] Very frustrated with Ceph! > > On Fri, Nov 1, 2013 at 11:12 PM, Sage Weil wrote: >> On Sat, 2 Nov 2013, Trivedi, Narendra wrote: >>> >>> Hi Sage, >>> >>> I believe I issued a "ceph-deploy install..." from the admin node as >>> per the documentation and that was almost ok as per the output of the >>> command below except sometimes there?s an error followed by an ?OK? >>> message (see the highlighted item in the red below). I eventually ran >>> into some permission issues but seems things went okay: > > Maybe what can be confusing here is that ceph-deploy interprets stderr as > ERROR logging level. Unfortunately, some tools will output normal informative > data to stderr when they are clearly not errors. > > stdout, on the other hand, is interpreted by ceph-deploy as DEBUG level, so > you will see logging at that level too. > > There is no way for ceph-deploy to tell if you are actually seeing errors > because the tool is in fact sending error messages or because it decided to > use stderr to send information that should go to stdout. > > > >> >> Hmm, the below output makes it look like it was successfully installed >> on >> node1 node2 and node3. Can you confirm that /etc/ceph exists on all >> three of those hosts? >> >> Oh, looking back at your original message, it looks like you are >> trying to create OSDs on /tmp/osd*. I would create directories like >> /ceph/osdo, /ceph/osd1, or similar. I believe you need to create the >> directories beforehand, too. (In a normal deployment, you are either >> feeding ceph raw disks (/dev/XXX) or an existing mount point on a >> dedicated disk you already configured and mounted.) >> >> sage >> >> >> > >>> >>> >>> [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ceph-deploy install >>> ceph-node1-mon-centos-6-4 ceph-node2-osd0-centos-6-4 >>> ceph-node3-osd1-centos-6-4 >>> >>> [ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy install >>> ceph-node1-mon-centos-6-4 ceph-node2-osd0-centos-6-4 >>> ceph-node3-osd1-centos-6-4 >>> >>> [ceph_deploy.install][DEBUG ] Installing stable version dumpling on >>> cluster ceph hosts ceph-node1-mon-centos-6-4 >>> ceph-node2-osd0-centos-6-4 >>> ceph-node3-osd1-centos-6-4 >>> >>> [ceph_deploy.install][DEBUG ] Detecting platform for host >>> ceph-node1-mon-centos-6-4 ... >>> >>> [ceph-node1-mon-centos-6-4][DEBUG ] connected to host: >>> ceph-node1-mon-centos-6-4 >>> >>> [ceph-node1-mon-centos-6-4][DEBUG ] detect platform information from >>> remote host >>> >>> [ceph-node1-mon-centos-6-4][DEBUG ] detect machine type >>> >>> [ceph_deploy.install][INFO ] Distro info: CentOS 6.4 Final >>> >>> [ceph-node1-mon-centos-6-4][INFO ] installing ceph on >>> ceph-node1-mon-centos-6-4 >>> >>> [ceph-node1-mon-centos-6-4][INFO ] adding EPEL repository >>> >>> [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo wget >>> http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch >>> .rpm >>> >>> [ceph-node1-mon-centos-6-4][ERROR ] --2013-11-01 19:51:20-- >>> http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch >>> .rpm >>> >>> [ceph-node1-mon-centos-6-4][ERROR ] Connecting to 10.12.132.208:8080... >>> connected. >>> >>> [ceph-node1-mon-centos-6-4][ERROR ] Proxy request sent, awaiting response... >>> 200 OK >>> >>> [ceph-node1-mon-centos-6-4][ERROR ] Length: 14540 (14K) >>> [application/x-rpm] >>> >>> [ceph-node1-mon-centos-6-4][ERROR ] Saving to: >>> `epel-release-6-8.noarch.rpm.2' >>> >>> [ceph-node1-mon-centos-6-4][ERROR ] >>> >>> [ceph-node1-mon-centos-6-4][ERROR ] 0K .. >>> 100% 4.79M=0.003s >>> >>> [ceph-node1-mon-centos-6-4][ERROR ] >>> >>> [ceph-node1-mon-centos-6-4][ERROR ] Last-modified header invalid -- >>> time-stamp ignored. >>> >>> [ceph-node1-mon-centos-6-4][ERROR ] 2013-11-01 19:52:20 (4.79 MB/s) - >>> `epel-release-6-8.noarch.rpm.2' saved [14540/14540] >>> >>> [ceph-node1-mon-centos-6-4][ERROR ] >>> >>> [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo rpm -Uvh >>> --replacepkgs epel-release-6*.rpm >>> >>> [ceph-node1-mon-centos-6-4][DEBUG ] Preparing... >>>
Re: [ceph-users] testing ceph
Pls can somebody help? Im getting this error.
ceph@CephAdmin:~$ ceph-deploy osd create
server1:sda:/dev/sdj1[ceph_deploy.cli][INFO ] Invoked (1.3):
/usr/bin/ceph-deploy osd create server1:sda:/dev/sdj1[ceph_deploy.osd][DEBUG ]
Preparing cluster ceph disks server1:/dev/sda:/dev/sdj1[server1][DEBUG ]
connected to host: server1[server1][DEBUG ] detect platform information from
remote host[server1][DEBUG ] detect machine type[ceph_deploy.osd][INFO ]
Distro info: Ubuntu 12.04 precise[ceph_deploy.osd][DEBUG ] Deploying osd to
server1[server1][DEBUG ] write cluster configuration to
/etc/ceph/{cluster}.conf[server1][INFO ] Running command: sudo udevadm trigger
--subsystem-match=block --action=add[ceph_deploy.osd][DEBUG ] Preparing host
server1 disk /dev/sda journal /dev/sdj1 activate True[server1][INFO ] Running
command: sudo ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sda
/dev/sdj1[server1][ERROR ] WARNING:ceph-disk:OSD will not be hot-swappable if
journal is not the same device as the osd data[server1][ERROR ] Could not
create partition 1 from 34 to 2047[server1][ERROR ] Error encountered; not
saving changes.[server1][ERROR ] ceph-disk: Error: Command '['sgdisk',
'--largest-new=1', '--change-name=1:ceph data',
'--partition-guid=1:d3ca8a92-7ba5-412e-abf5-06af958b788d',
'--typecode=1:89c57f98-2fe5-4dc0-89c1-f3ad0ceff2be', '--', '/dev/sda']'
returned non-zero exit status 4[server1][ERROR ] Traceback (most recent call
last):[server1][ERROR ] File
"/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/process.py", line 68,
in run[server1][ERROR ] reporting(conn, result, timeout)[server1][ERROR ]
File "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/log.py", line 13,
in reporting[server1][ERROR ] received =
result.receive(timeout)[server1][ERROR ] File
"/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py",
line 455, in receive[server1][ERROR ] raise self._getremoteerror() or
EOFError()[server1][ERROR ] RemoteError: Traceback (most recent call
last):[server1][ERROR ] File "", line 806, in
executetask[server1][ERROR ] File "", line 35, in _remote_run[server1][ERROR
] RuntimeError: command returned non-zero exit status: 1[server1][ERROR
][server1][ERROR ][ceph_deploy.osd][ERROR ] Failed to execute command:
ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sda
/dev/sdj1[ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs
> Date: Thu, 31 Oct 2013 10:55:56 +
> From: joao.l...@inktank.com
> To: charlesboy...@hotmail.com; ceph-de...@vger.kernel.org
> Subject: Re: testing ceph
>
> On 10/31/2013 04:54 AM, charles L wrote:
> > Hi,
> > Pls is this a good setup for a production environment test of ceph? My
> > focus is on the SSD ... should it be partitioned(sdf1,2 ,3,4) and shared by
> > the four OSDs on a host? or is this a better configuration for the SSD to
> > be just one partition(sdf1) while all osd uses that one partition?
> > my setup:
> > - 6 Servers with one 250gb boot disk for OS(sda),
> > four-2Tb Disks each for the OSDs i.e Total disks = 6x4 = 24 disks (sdb -sde)
> > and one-60GB SSD for Osd Journal(sdf).
> > -RAM = 32GB on each server with 2 GB network link.
> > hostname for servers: Server1 -Server6
>
> Charles,
>
> What you are describing on the ceph.conf below is definitely not a good
> idea. If you really want to use just one SSD and share it across
> multiple OSDs, then you have two possible approaches:
>
> - partition that disk and assign a *different* partition to each OSD; or
> - keep only one partition, format it with some filesystem, and assign a
> *different* journal file within that fs to each OSD.
>
> What you are describing has you using the same partition for all OSDs.
> This will likely create issues due to multiple OSDs writing and reading
> from a single journal. TBH I'm not familiar enough with the journal
> mechanism to know whether the OSDs will detect that situation.
>
>-Joao
>
> >
> > [osd.0]
> > host = server1
> > devs = /dev/sdb
> > osd journal = /dev/sdf1
> > [osd.1]
> > host = server1
> > devs = /dev/sdc
> > osd journal = /dev/sdf2
> >
> > [osd.3]
> > host = server1
> > devs = /dev/sdd
> > osd journal = /dev/sdf2
> >
> > [osd.4]
> > host = server1
> > devs = /dev/sde
> > osd journal = /dev/sdf2
> > [osd.5]
> > host = server2
> > devs = /dev/sdb
> > osd journal = /dev/sdf2
> > ...
> > [osd.23]
> > host = server6
> > devs = /dev/sde
> > osd journal = /dev/sdf2
> >
> > Thanks. --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majord...@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
>
>
> --
> Joao Eduardo Luis
> Software Engineer | http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@l
Re: [ceph-users] testing ceph
Is disk sda on server1 empty or does it contain already a partition?
On Nov 4, 2013, at 5:25 PM, charles L wrote:
>
> Pls can somebody help? Im getting this error.
>
> ceph@CephAdmin:~$ ceph-deploy osd create server1:sda:/dev/sdj1
> [ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy osd create
> server1:sda:/dev/sdj1
> [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks
> server1:/dev/sda:/dev/sdj1
> [server1][DEBUG ] connected to host: server1
> [server1][DEBUG ] detect platform information from remote host
> [server1][DEBUG ] detect machine type
> [ceph_deploy.osd][INFO ] Distro info: Ubuntu 12.04 precise
> [ceph_deploy.osd][DEBUG ] Deploying osd to server1
> [server1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
> [server1][INFO ] Running command: sudo udevadm trigger
> --subsystem-match=block --action=add
> [ceph_deploy.osd][DEBUG ] Preparing host server1 disk /dev/sda journal
> /dev/sdj1 activate True
> [server1][INFO ] Running command: sudo ceph-disk-prepare --fs-type xfs
> --cluster ceph -- /dev/sda /dev/sdj1
> [server1][ERROR ] WARNING:ceph-disk:OSD will not be hot-swappable if journal
> is not the same device as the osd data
> [server1][ERROR ] Could not create partition 1 from 34 to 2047
> [server1][ERROR ] Error encountered; not saving changes.
> [server1][ERROR ] ceph-disk: Error: Command '['sgdisk', '--largest-new=1',
> '--change-name=1:ceph data',
> '--partition-guid=1:d3ca8a92-7ba5-412e-abf5-06af958b788d',
> '--typecode=1:89c57f98-2fe5-4dc0-89c1-f3ad0ceff2be', '--', '/dev/sda']'
> returned non-zero exit status 4
> [server1][ERROR ] Traceback (most recent call last):
> [server1][ERROR ] File
> "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/process.py", line
> 68, in run
> [server1][ERROR ] reporting(conn, result, timeout)
> [server1][ERROR ] File
> "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/log.py", line 13, in
> reporting
> [server1][ERROR ] received = result.receive(timeout)
> [server1][ERROR ] File
> "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py",
> line 455, in receive
> [server1][ERROR ] raise self._getremoteerror() or EOFError()
> [server1][ERROR ] RemoteError: Traceback (most recent call last):
> [server1][ERROR ] File "", line 806, in executetask
> [server1][ERROR ] File "", line 35, in _remote_run
> [server1][ERROR ] RuntimeError: command returned non-zero exit status: 1
> [server1][ERROR ]
> [server1][ERROR ]
> [ceph_deploy.osd][ERROR ] Failed to execute command: ceph-disk-prepare
> --fs-type xfs --cluster ceph -- /dev/sda /dev/sdj1
> [ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs
>
>
>
>
>
>
> > Date: Thu, 31 Oct 2013 10:55:56 +
> > From: joao.l...@inktank.com
> > To: charlesboy...@hotmail.com; ceph-de...@vger.kernel.org
> > Subject: Re: testing ceph
> >
> > On 10/31/2013 04:54 AM, charles L wrote:
> > > Hi,
> > > Pls is this a good setup for a production environment test of ceph? My
> > > focus is on the SSD ... should it be partitioned(sdf1,2 ,3,4) and shared
> > > by the four OSDs on a host? or is this a better configuration for the SSD
> > > to be just one partition(sdf1) while all osd uses that one partition?
> > > my setup:
> > > - 6 Servers with one 250gb boot disk for OS(sda),
> > > four-2Tb Disks each for the OSDs i.e Total disks = 6x4 = 24 disks (sdb
> > > -sde)
> > > and one-60GB SSD for Osd Journal(sdf).
> > > -RAM = 32GB on each server with 2 GB network link.
> > > hostname for servers: Server1 -Server6
> >
> > Charles,
> >
> > What you are describing on the ceph.conf below is definitely not a good
> > idea. If you really want to use just one SSD and share it across
> > multiple OSDs, then you have two possible approaches:
> >
> > - partition that disk and assign a *different* partition to each OSD; or
> > - keep only one partition, format it with some filesystem, and assign a
> > *different* journal file within that fs to each OSD.
> >
> > What you are describing has you using the same partition for all OSDs.
> > This will likely create issues due to multiple OSDs writing and reading
> > from a single journal. TBH I'm not familiar enough with the journal
> > mechanism to know whether the OSDs will detect that situation.
> >
> > -Joao
> >
> > >
> > > [osd.0]
> > > host = server1
> > > devs = /dev/sdb
> > > osd journal = /dev/sdf1
> > > [osd.1]
> > > host = server1
> > > devs = /dev/sdc
> > > osd journal = /dev/sdf2
> > >
> > > [osd.3]
> > > host = server1
> > > devs = /dev/sdd
> > > osd journal = /dev/sdf2
> > >
> > > [osd.4]
> > > host = server1
> > > devs = /dev/sde
> > > osd journal = /dev/sdf2
> > > [osd.5]
> > > host = server2
> > > devs = /dev/sdb
> > > osd journal = /dev/sdf2
> > > ...
> > > [osd.23]
> > > host = server6
> > > devs = /dev/sde
> > > osd journal = /dev/sdf2
> > >
> > > Thanks. --
> > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > >
Re: [ceph-users] testing ceph
Bingo! A lot of people are getting this dreadful GenericErro and Failed to
create 1 OSD. Does anyone know why despite /etc/ceph being there on each node?
Also, FYI purgedata on multiple nodes doesn't work sometime i.e. it says it is
uninstalled ceph and removed /etc/ceph from all nodes but they are there on all
nodes except the first one (i.e. the first argument to the purgedata command ).
Hence sometimes, I have to issue purgedata to individual nodes.
From: ceph-users-boun...@lists.ceph.com
[mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of charles L
Sent: Monday, November 04, 2013 9:26 AM
To: ceph-de...@vger.kernel.org; ceph-us...@ceph.com
Subject: Re: [ceph-users] testing ceph
Pls can somebody help? Im getting this error.
ceph@CephAdmin:~$ ceph-deploy osd create server1:sda:/dev/sdj1
[ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy osd create
server1:sda:/dev/sdj1
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks
server1:/dev/sda:/dev/sdj1
[server1][DEBUG ] connected to host: server1
[server1][DEBUG ] detect platform information from remote host
[server1][DEBUG ] detect machine type
[ceph_deploy.osd][INFO ] Distro info: Ubuntu 12.04 precise
[ceph_deploy.osd][DEBUG ] Deploying osd to server1
[server1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[server1][INFO ] Running command: sudo udevadm trigger --subsystem-match=block
--action=add
[ceph_deploy.osd][DEBUG ] Preparing host server1 disk /dev/sda journal
/dev/sdj1 activate True
[server1][INFO ] Running command: sudo ceph-disk-prepare --fs-type xfs
--cluster ceph -- /dev/sda /dev/sdj1
[server1][ERROR ] WARNING:ceph-disk:OSD will not be hot-swappable if journal is
not the same device as the osd data
[server1][ERROR ] Could not create partition 1 from 34 to 2047
[server1][ERROR ] Error encountered; not saving changes.
[server1][ERROR ] ceph-disk: Error: Command '['sgdisk', '--largest-new=1',
'--change-name=1:ceph data',
'--partition-guid=1:d3ca8a92-7ba5-412e-abf5-06af958b788d',
'--typecode=1:89c57f98-2fe5-4dc0-89c1-f3ad0ceff2be', '--', '/dev/sda']'
returned non-zero exit status 4
[server1][ERROR ] Traceback (most recent call last):
[server1][ERROR ] File
"/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/process.py", line 68,
in run
[server1][ERROR ] reporting(conn, result, timeout)
[server1][ERROR ] File
"/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/log.py", line 13, in
reporting
[server1][ERROR ] received = result.receive(timeout)
[server1][ERROR ] File
"/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py",
line 455, in receive
[server1][ERROR ] raise self._getremoteerror() or EOFError()
[server1][ERROR ] RemoteError: Traceback (most recent call last):
[server1][ERROR ] File "", line 806, in executetask
[server1][ERROR ] File "", line 35, in _remote_run
[server1][ERROR ] RuntimeError: command returned non-zero exit status: 1
[server1][ERROR ]
[server1][ERROR ]
[ceph_deploy.osd][ERROR ] Failed to execute command: ceph-disk-prepare
--fs-type xfs --cluster ceph -- /dev/sda /dev/sdj1
[ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs
> Date: Thu, 31 Oct 2013 10:55:56 +
> From: joao.l...@inktank.com
> To: charlesboy...@hotmail.com;
> ceph-de...@vger.kernel.org
> Subject: Re: testing ceph
>
> On 10/31/2013 04:54 AM, charles L wrote:
> > Hi,
> > Pls is this a good setup for a production environment test of ceph? My
> > focus is on the SSD ... should it be partitioned(sdf1,2 ,3,4) and shared by
> > the four OSDs on a host? or is this a better configuration for the SSD to
> > be just one partition(sdf1) while all osd uses that one partition?
> > my setup:
> > - 6 Servers with one 250gb boot disk for OS(sda),
> > four-2Tb Disks each for the OSDs i.e Total disks = 6x4 = 24 disks (sdb -sde)
> > and one-60GB SSD for Osd Journal(sdf).
> > -RAM = 32GB on each server with 2 GB network link.
> > hostname for servers: Server1 -Server6
>
> Charles,
>
> What you are describing on the ceph.conf below is definitely not a good
> idea. If you really want to use just one SSD and share it across
> multiple OSDs, then you have two possible approaches:
>
> - partition that disk and assign a *different* partition to each OSD; or
> - keep only one partition, format it with some filesystem, and assign a
> *different* journal file within that fs to each OSD.
>
> What you are describing has you using the same partition for all OSDs.
> This will likely create issues due to multiple OSDs writing and reading
> from a single journal. TBH I'm not familiar enough with the journal
> mechanism to know whether the OSDs will detect that situation.
>
> -Joao
>
> >
> > [osd.0]
> > host = server1
> > devs = /dev/sdb
> > osd journal = /dev/sdf1
> > [osd.1]
> > host = server1
> > devs = /dev/sdc
> > osd journal = /dev/sdf2
> >
> > [osd.3
Re: [ceph-users] testing ceph
On Mon, Nov 4, 2013 at 10:56 AM, Trivedi, Narendra
wrote:
> Bingo! A lot of people are getting this dreadful GenericErro and Failed to
> create 1 OSD. Does anyone know why despite /etc/ceph being there on each
> node?
/etc/ceph is created by installing ceph on a node, and purgedata will
remove the contents of /etc/ceph/
and not the actual directory in the latest (1.3) version.
Also, FYI purgedata on multiple nodes doesn’t work sometime i.e. it
> says it is uninstalled ceph and removed /etc/ceph from all nodes but they
> are there on all nodes except the first one (i.e. the first argument to the
> purgedata command ). Hence sometimes, I have to issue purgedata to
> individual nodes.
That does sound unexpected behavior from ceph-deploy. Can you share
some logs that demonstrate
this? Like I said, /etc/ceph is actually no longer removed in the
latest version, just the contents.
And you say "sometimes" as in, this doesn't happen consistently? Or do
you mean something else?
Again, log output and how you got there would be useful to try and
determine what is going on.
>
>
>
> From: ceph-users-boun...@lists.ceph.com
> [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of charles L
> Sent: Monday, November 04, 2013 9:26 AM
> To: ceph-de...@vger.kernel.org; ceph-us...@ceph.com
>
>
> Subject: Re: [ceph-users] testing ceph
>
>
>
>
>
> Pls can somebody help? Im getting this error.
>
>
>
> ceph@CephAdmin:~$ ceph-deploy osd create server1:sda:/dev/sdj1
>
> [ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy osd create
> server1:sda:/dev/sdj1
>
> [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks
> server1:/dev/sda:/dev/sdj1
>
> [server1][DEBUG ] connected to host: server1
>
> [server1][DEBUG ] detect platform information from remote host
>
> [server1][DEBUG ] detect machine type
>
> [ceph_deploy.osd][INFO ] Distro info: Ubuntu 12.04 precise
>
> [ceph_deploy.osd][DEBUG ] Deploying osd to server1
>
> [server1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
>
> [server1][INFO ] Running command: sudo udevadm trigger
> --subsystem-match=block --action=add
>
> [ceph_deploy.osd][DEBUG ] Preparing host server1 disk /dev/sda journal
> /dev/sdj1 activate True
>
> [server1][INFO ] Running command: sudo ceph-disk-prepare --fs-type xfs
> --cluster ceph -- /dev/sda /dev/sdj1
>
> [server1][ERROR ] WARNING:ceph-disk:OSD will not be hot-swappable if journal
> is not the same device as the osd data
>
> [server1][ERROR ] Could not create partition 1 from 34 to 2047
>
> [server1][ERROR ] Error encountered; not saving changes.
>
> [server1][ERROR ] ceph-disk: Error: Command '['sgdisk', '--largest-new=1',
> '--change-name=1:ceph data',
> '--partition-guid=1:d3ca8a92-7ba5-412e-abf5-06af958b788d',
> '--typecode=1:89c57f98-2fe5-4dc0-89c1-f3ad0ceff2be', '--', '/dev/sda']'
> returned non-zero exit status 4
>
> [server1][ERROR ] Traceback (most recent call last):
>
> [server1][ERROR ] File
> "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/process.py", line
> 68, in run
>
> [server1][ERROR ] reporting(conn, result, timeout)
>
> [server1][ERROR ] File
> "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/log.py", line 13,
> in reporting
>
> [server1][ERROR ] received = result.receive(timeout)
>
> [server1][ERROR ] File
> "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py",
> line 455, in receive
>
> [server1][ERROR ] raise self._getremoteerror() or EOFError()
>
> [server1][ERROR ] RemoteError: Traceback (most recent call last):
>
> [server1][ERROR ] File "", line 806, in executetask
>
> [server1][ERROR ] File "", line 35, in _remote_run
>
> [server1][ERROR ] RuntimeError: command returned non-zero exit status: 1
>
> [server1][ERROR ]
>
> [server1][ERROR ]
>
> [ceph_deploy.osd][ERROR ] Failed to execute command: ceph-disk-prepare
> --fs-type xfs --cluster ceph -- /dev/sda /dev/sdj1
>
> [ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs
>
>
>
>
>
>
>
>
>
>
>
>
>
>> Date: Thu, 31 Oct 2013 10:55:56 +
>> From: joao.l...@inktank.com
>> To: charlesboy...@hotmail.com; ceph-de...@vger.kernel.org
>> Subject: Re: testing ceph
>>
>> On 10/31/2013 04:54 AM, charles L wrote:
>> > Hi,
>> > Pls is this a good setup for a production environment test of ceph? My
>> > focus is on the SSD ... should it be partitioned(sdf1,2 ,3,4) and shared by
>> > the four OSDs on a host? or is this a better configuration for the SSD to
>> > be
>> > just one partition(sdf1) while all osd uses that one partition?
>> > my setup:
>> > - 6 Servers with one 250gb boot disk for OS(sda),
>> > four-2Tb Disks each for the OSDs i.e Total disks = 6x4 = 24 disks (sdb
>> > -sde)
>> > and one-60GB SSD for Osd Journal(sdf).
>> > -RAM = 32GB on each server with 2 GB network link.
>> > hostname for servers: Server1 -Server6
>>
>> Charles,
>>
>> What you are describing on the ceph.conf below is definitely not a good
>> idea. If you really want to use just one SSD and share it
Re: [ceph-users] Very frustrated with Ceph!
On Mon, Nov 4, 2013 at 12:25 PM, Gruher, Joseph R wrote: > Could these problems be caused by running a purgedata but not a purge? It could be, I am not clear on what the expectation was for just doing purgedata without a purge. > Purgedata removes /etc/ceph but without the purge ceph is still installed, > then ceph-deploy install detects ceph as already installed and does not > (re)create /etc/ceph? ceph-deploy will not create directories for you, that is left to the ceph install process, and just to be clear, the latest ceph-deploy version (1.3) does not remote /etc/ceph, just the contents. > > > > [ceph-node2-osd0-centos-6-4][DEBUG ] Package ceph-0.67.4-0.el6.x86_64 > already installed and latest version > > > > I wonder if you ran a purge and a purgedata if you might have better luck. > That always works for me. > > > > From: ceph-users-boun...@lists.ceph.com > [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Trivedi, Narendra > Sent: Saturday, November 02, 2013 10:42 PM > To: Sage Weil > Cc: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] Very frustrated with Ceph! > > > > Thanks a lot Sage for your help :-). > > > > I started from scratch: See the commands and output below: > > > > 1) First of all, all the nodes did have but in order to start from scratch I > removed /etc/ceph from each node. > > > > 2) I issued a ceph-deploy purgedata to each node from the admin node. This > threw error towards the end. I assuming since I manually removed /etc/ceph > from nodes and hence rm command fails: > > > > [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ceph-deploy purgedata > ceph-node1-mon-centos-6-4 ceph-node2-osd0-centos-6-4 > ceph-node3-osd1-centos-6-4 > > [ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy purgedata > ceph-node1-mon-centos-6-4 ceph-node2-osd0-centos-6-4 > ceph-node3-osd1-centos-6-4 > > [ceph_deploy.install][DEBUG ] Purging data from cluster ceph hosts > ceph-node1-mon-centos-6-4 ceph-node2-osd0-centos-6-4 > ceph-node3-osd1-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] connected to host: > ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node1-mon-centos-6-4][DEBUG ] detect machine type > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo which ceph > > [ceph-node2-osd0-centos-6-4][DEBUG ] connected to host: > ceph-node2-osd0-centos-6-4 > > [ceph-node2-osd0-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node2-osd0-centos-6-4][DEBUG ] detect machine type > > [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo which ceph > > [ceph-node3-osd1-centos-6-4][DEBUG ] connected to host: > ceph-node3-osd1-centos-6-4 > > [ceph-node3-osd1-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node3-osd1-centos-6-4][DEBUG ] detect machine type > > [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo which ceph > > ceph is still installed on: ['ceph-node1-mon-centos-6-4', > 'ceph-node2-osd0-centos-6-4', 'ceph-node3-osd1-centos-6-4'] > > Continue (y/n)y > > [ceph-node1-mon-centos-6-4][DEBUG ] connected to host: > ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node1-mon-centos-6-4][DEBUG ] detect machine type > > [ceph_deploy.install][INFO ] Distro info: CentOS 6.4 Final > > [ceph-node1-mon-centos-6-4][INFO ] purging data on > ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo rm -rf > --one-file-system -- /var/lib/ceph > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo rm -rf > --one-file-system -- /etc/ceph/* > > [ceph-node2-osd0-centos-6-4][DEBUG ] connected to host: > ceph-node2-osd0-centos-6-4 > > [ceph-node2-osd0-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node2-osd0-centos-6-4][DEBUG ] detect machine type > > [ceph_deploy.install][INFO ] Distro info: CentOS 6.4 Final > > [ceph-node2-osd0-centos-6-4][INFO ] purging data on > ceph-node2-osd0-centos-6-4 > > [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo rm -rf > --one-file-system -- /var/lib/ceph > > [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo rm -rf > --one-file-system -- /etc/ceph/* > > Exception in thread Thread-1 (most likely raised during interpreter > shutdown): > > Traceback (most recent call last): > > File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner > > File "", line 89, in run > > : 140730692818688)> > > [ceph-node3-osd1-centos-6-4][DEBUG ] connected to host: > ceph-node3-osd1-centos-6-4 > > [ceph-node3-osd1-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node3-osd1-centos-6-4][DEBUG ] detect machine type > > [ceph_deploy.install][INFO ] Distro info: CentOS 6.4 Final > > [ceph-node3-osd1-centos-6-4][INFO ] purging data on > ceph-node3-osd1-centos-6-4 > > [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo rm -rf
Re: [ceph-users] rbd on ubuntu 12.04 LTS
On Mon, Nov 4, 2013 at 12:13 AM, Fuchs, Andreas (SwissTXT)
wrote:
> I tryed with:
> ceph osd crush tunables default
> ceph osd crush tunables argonaut
>
> while the command runs without error, I still get the feature set mismatch
> error whe I try to mount
> do I have to restart some service?
Ah, looking more closely it seems the feature mismatch you're getting
is actually the "HASHPSPOOL" feature bit. I don't think that should
have been enabled on Dumpling, but you can unset it on a pool basis
("ceph osd pool unset hashpspool", I believe). I don't think
you'll need to restart anything, but it's possible.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] rgw bucket creation fails
On Mon, Nov 4, 2013 at 6:40 AM, Corin Langosch wrote: > Hi, > > using ceph 0.67.4 I followed http://ceph.com/docs/master/radosgw/. I can > connect using s3cmd (test configuration succeeds), so the user credentials > and everything else seems to be running as it should. But when doing a > "s3cmd mb s3://test" the radosgw returns a "405 Method Not Allowed" > (connection captured with wireshark). The radosgw logs don't contain anythin > related to this. > > What am I doing wrong? > This was answered off list on irc, but for the sake of completeness I'll answer here too. The issue is that s3cmd uses a virtual bucket host name. E.g., instead of http:///bucket, it sends request to http://., so in order for the gateway to identify which bucket is being used, it needs to be configured with the host name. So adding the following under the relevant section in your ceph.conf should solve the issue: rgw dns name = Yehuda ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] rgw bucket creation fails
04.11.2013 19:56, schrieb Yehuda Sadeh: This was answered off list on irc, but for the sake of completeness I'll answer here too. The issue is that s3cmd uses a virtual bucket host name. E.g., instead of http:///bucket, it sends request to http://., so in order for the gateway to identify which bucket is being used, it needs to be configured with the host name. So adding the following under the relevant section in your ceph.conf should solve the issue: rgw dns name = Yes this did the trick! One also has to remember to add a "ServerAlias *.example.com" to his apache vhost settings. Thanks for your quick help! Corin ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] radosgw fails to start
Sorry to bump this, but does anyone have any idea what could be wrong here?
To resummarize, radosgw fails to start. Debug output seems to indicate it is
complaining about the keyring, but the keyring is present and readable, and
other Ceph functions which require the keyring can success. So why can't
radosgw start? Details below.
Thanks!
>-Original Message-
>From: Gruher, Joseph R
>Sent: Friday, November 01, 2013 11:50 AM
>To: Gruher, Joseph R
>Subject: RE: radosgw fails to start
>
>>Adding some debug arguments has generated output which I believe
>>indicates the problem is my keyring is missing, but the keyring seems
>>to be here. Why would this complain about the keyring and fail to start?
>>
>>[ceph@joceph08 ceph]$ sudo /usr/bin/radosgw -d --debug-rgw 20 --debug-
>>ms 1 start
>>2013-11-01 10:59:47.015332 7f83978e4820 0 ceph version 0.67.4
>>(ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7), process radosgw, pid 18760
>>2013-11-01 10:59:47.015338 7f83978e4820 -1 WARNING: libcurl doesn't
>>support
>>curl_multi_wait()
>>2013-11-01 10:59:47.015340 7f83978e4820 -1 WARNING: cross zone / region
>>transfer performance may be affected
>>2013-11-01 10:59:47.018707 7f83978e4820 1 -- :/0 messenger.start
>>2013-11-01 10:59:47.018773 7f83978e4820 -1 monclient(hunting): ERROR:
>>missing keyring, cannot use cephx for authentication
>>2013-11-01 10:59:47.018774 7f83978e4820 0 librados: client.admin
>>initialization error (2) No such file or directory
>>2013-11-01 10:59:47.018788 7f83978e4820 1 -- :/1018760 mark_down_all
>>2013-11-01 10:59:47.018932 7f83978e4820 1 -- :/1018760 shutdown
>complete.
>>2013-11-01 10:59:47.018967 7f83978e4820 -1 Couldn't init storage
>>provider
>>(RADOS)
>>
>>[ceph@joceph08 ceph]$ sudo service ceph-radosgw status
>/usr/bin/radosgw
>>is not running.
>>
>>[ceph@joceph08 ceph]$ pwd
>>/etc/ceph
>>
>>[ceph@joceph08 ceph]$ ls
>>ceph.client.admin.keyring ceph.conf keyring.radosgw.gateway rbdmap
>>
>>[ceph@joceph08 ceph]$ cat ceph.client.admin.keyring [client.admin]
>> key = AQCYyHJSCFH3BBAA472q80qrAiIIVbvJfK/47A==
>>
>>[ceph@joceph08 ceph]$ cat keyring.radosgw.gateway
>>[client.radosgw.gateway]
>> key = AQBh6nNS0Cu3HxAAMxLsbEYZ3pEbwEBajQb1WA==
>> caps mon = "allow rw"
>> caps osd = "allow rwx"
>>
>>[ceph@joceph08 ceph]$ cat ceph.conf
>>[client.radosgw.joceph08]
>>host = joceph08
>>log_file = /var/log/ceph/radosgw.log
>>keyring = /etc/ceph/keyring.radosgw.gateway rgw_socket_path =
>>/tmp/radosgw.sock
>>
>>[global]
>>auth_service_required = cephx
>>filestore_xattr_use_omap = true
>>auth_client_required = cephx
>>auth_cluster_required = cephx
>>mon_host = 10.23.37.142,10.23.37.145,10.23.37.161,10.23.37.165
>>osd_journal_size = 1024
>>mon_initial_members = joceph01, joceph02, joceph03, joceph04 fsid =
>>74d808db-aaa7-41d2-8a84-7d590327a3c7
>
>By the way, I can run other commands on the node which I think must require
>the keyring. they succeed.
>
>[ceph@joceph08 ceph]$ sudo /usr/bin/radosgw -d -c /etc/ceph/ceph.conf --
>debug-rgw 20 --debug-ms 1 start
>2013-11-01 11:45:07.935483 7ff2e2f11820 0 ceph version 0.67.4
>(ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7), process radosgw, pid 19265
>2013-11-01 11:45:07.935488 7ff2e2f11820 -1 WARNING: libcurl doesn't support
>curl_multi_wait()
>2013-11-01 11:45:07.935489 7ff2e2f11820 -1 WARNING: cross zone / region
>transfer performance may be affected
>2013-11-01 11:45:07.938719 7ff2e2f11820 1 -- :/0 messenger.start
>2013-11-01 11:45:07.938817 7ff2e2f11820 -1 monclient(hunting): ERROR:
>missing keyring, cannot use cephx for authentication
>2013-11-01 11:45:07.938818 7ff2e2f11820 0 librados: client.admin
>initialization
>error (2) No such file or directory
>2013-11-01 11:45:07.938832 7ff2e2f11820 1 -- :/1019265 mark_down_all
>2013-11-01 11:45:07.939150 7ff2e2f11820 1 -- :/1019265 shutdown complete.
>2013-11-01 11:45:07.939219 7ff2e2f11820 -1 Couldn't init storage provider
>(RADOS)
>
>[ceph@joceph08 ceph]$ rados df
>pool name category KB objects clones
>degraded unfound
>rdrd KB wrwr KB
>data- 000
> 0 0000
>0
>metadata- 000
> 0 0000
>0
>rbd - 000
> 0 0000
>0
> total used 6306480
> total avail11714822792
> total space11715453440
>
>[ceph@joceph08 ceph]$ ceph status
> cluster 74d808db-aaa7-41d2-8a84-7d590327a3c7
> health HEALTH_OK
> monmap e1: 4 mons at
>{joceph01=10.23.37.142:6789/0,joceph02=10.23.37.145:6789/0,joceph03=10.2
>3.37.161:6789/0,joceph04=10.23.37.165:6789/0}, election epoch 8, quorum
>0,1,2,3 joceph01,joceph02,joceph03,joceph04
> osdmap e88: 16 osds: 16 up,
Re: [ceph-users] radosgw fails to start
Not sure why you're able to run the 'rados' and 'ceph' command, and
not 'radosgw', just note that the former two don't connect to the
osds, whereas the latter does, so it might fail on a different level.
You're using the default client.admin as the user for radosgw, but
your ceph.conf file doesn't have a section for it and all the relevant
configurables are under client.radosgw.gateway. Try fixing that first.
Yehuda
On Mon, Nov 4, 2013 at 12:30 PM, Gruher, Joseph R
wrote:
> Sorry to bump this, but does anyone have any idea what could be wrong here?
>
> To resummarize, radosgw fails to start. Debug output seems to indicate it is
> complaining about the keyring, but the keyring is present and readable, and
> other Ceph functions which require the keyring can success. So why can't
> radosgw start? Details below.
>
...
>>>2013-11-01 10:59:47.018774 7f83978e4820 0 librados: client.admin
...
>>>[ceph@joceph08 ceph]$ pwd
>>>/etc/ceph
>>>
>>>[ceph@joceph08 ceph]$ ls
>>>ceph.client.admin.keyring ceph.conf keyring.radosgw.gateway rbdmap
>>>
>>>[ceph@joceph08 ceph]$ cat ceph.client.admin.keyring [client.admin]
>>>key = AQCYyHJSCFH3BBAA472q80qrAiIIVbvJfK/47A==
>>>
>>>[ceph@joceph08 ceph]$ cat keyring.radosgw.gateway
>>>[client.radosgw.gateway]
>>>key = AQBh6nNS0Cu3HxAAMxLsbEYZ3pEbwEBajQb1WA==
>>>caps mon = "allow rw"
>>>caps osd = "allow rwx"
>>>
>>>[ceph@joceph08 ceph]$ cat ceph.conf
>>>[client.radosgw.joceph08]
>>>host = joceph08
>>>log_file = /var/log/ceph/radosgw.log
>>>keyring = /etc/ceph/keyring.radosgw.gateway rgw_socket_path =
>>>/tmp/radosgw.sock
>>>
>>>[global]
>>>auth_service_required = cephx
>>>filestore_xattr_use_omap = true
>>>auth_client_required = cephx
>>>auth_cluster_required = cephx
>>>mon_host = 10.23.37.142,10.23.37.145,10.23.37.161,10.23.37.165
>>>osd_journal_size = 1024
>>>mon_initial_members = joceph01, joceph02, joceph03, joceph04 fsid =
>>>74d808db-aaa7-41d2-8a84-7d590327a3c7
>>
>>By the way, I can run other commands on the node which I think must require
>>the keyring. they succeed.
>>
>>[ceph@joceph08 ceph]$ sudo /usr/bin/radosgw -d -c /etc/ceph/ceph.conf --
>>debug-rgw 20 --debug-ms 1 start
>>2013-11-01 11:45:07.935483 7ff2e2f11820 0 ceph version 0.67.4
>>(ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7), process radosgw, pid 19265
>>2013-11-01 11:45:07.935488 7ff2e2f11820 -1 WARNING: libcurl doesn't support
>>curl_multi_wait()
>>2013-11-01 11:45:07.935489 7ff2e2f11820 -1 WARNING: cross zone / region
>>transfer performance may be affected
>>2013-11-01 11:45:07.938719 7ff2e2f11820 1 -- :/0 messenger.start
>>2013-11-01 11:45:07.938817 7ff2e2f11820 -1 monclient(hunting): ERROR:
>>missing keyring, cannot use cephx for authentication
>>2013-11-01 11:45:07.938818 7ff2e2f11820 0 librados: client.admin
>>initialization
>>error (2) No such file or directory
>>2013-11-01 11:45:07.938832 7ff2e2f11820 1 -- :/1019265 mark_down_all
>>2013-11-01 11:45:07.939150 7ff2e2f11820 1 -- :/1019265 shutdown complete.
>>2013-11-01 11:45:07.939219 7ff2e2f11820 -1 Couldn't init storage provider
>>(RADOS)
>>
>>[ceph@joceph08 ceph]$ rados df
>>pool name category KB objects clones
>>degraded unfound
>>rdrd KB wrwr KB
>>data- 000
>>0 0000
>>0
>>metadata- 000
>>0 0000
>>0
>>rbd - 000
>>0 0000
>>0
>> total used 6306480
>> total avail11714822792
>> total space11715453440
>>
>>[ceph@joceph08 ceph]$ ceph status
>> cluster 74d808db-aaa7-41d2-8a84-7d590327a3c7
>> health HEALTH_OK
>> monmap e1: 4 mons at
>>{joceph01=10.23.37.142:6789/0,joceph02=10.23.37.145:6789/0,joceph03=10.2
>>3.37.161:6789/0,joceph04=10.23.37.165:6789/0}, election epoch 8, quorum
>>0,1,2,3 joceph01,joceph02,joceph03,joceph04
>> osdmap e88: 16 osds: 16 up, 16 in
>>pgmap v1402: 2400 pgs: 2400 active+clean; 0 bytes data, 615 MB used, 11172
>>GB / 11172 GB avail
>> mdsmap e1: 0/0/1 up
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] radosgw fails to start
>-Original Message- >From: Yehuda Sadeh [mailto:yeh...@inktank.com] >Sent: Monday, November 04, 2013 12:40 PM >To: Gruher, Joseph R >Cc: ceph-users@lists.ceph.com >Subject: Re: [ceph-users] radosgw fails to start > >Not sure why you're able to run the 'rados' and 'ceph' command, and not >'radosgw', just note that the former two don't connect to the osds, whereas >the latter does, so it might fail on a different level. >You're using the default client.admin as the user for radosgw, but your >ceph.conf file doesn't have a section for it and all the relevant configurables >are under client.radosgw.gateway. Try fixing that first. > >Yehuda > Thanks for the hint. Adding the section below seems to have addressed the problem. For some reason I didn't have to do this on my previous cluster but it seems to need it here. [client.admin] keyring = /etc/ceph/ceph.client.admin.keyring Now I am failing with a new problem, probably something to do with how I set up Apache, I think, this seems to be some kind of FastCGI error: 2013-11-04 13:05:48.354547 7f1cd6f5d820 0 ERROR: FCGX_Accept_r returned -88 Full output: http://pastebin.com/gyhQnrgP ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] radosgw fails to start
On Mon, Nov 4, 2013 at 1:12 PM, Gruher, Joseph R wrote: >>-Original Message- >>From: Yehuda Sadeh [mailto:yeh...@inktank.com] >>Sent: Monday, November 04, 2013 12:40 PM >>To: Gruher, Joseph R >>Cc: ceph-users@lists.ceph.com >>Subject: Re: [ceph-users] radosgw fails to start >> >>Not sure why you're able to run the 'rados' and 'ceph' command, and not >>'radosgw', just note that the former two don't connect to the osds, whereas >>the latter does, so it might fail on a different level. >>You're using the default client.admin as the user for radosgw, but your >>ceph.conf file doesn't have a section for it and all the relevant >>configurables >>are under client.radosgw.gateway. Try fixing that first. >> >>Yehuda >> > > Thanks for the hint. Adding the section below seems to have addressed the > problem. For some reason I didn't have to do this on my previous cluster but > it seems to need it here. > > [client.admin] > keyring = /etc/ceph/ceph.client.admin.keyring > > Now I am failing with a new problem, probably something to do with how I set > up Apache, I think, this seems to be some kind of FastCGI error: > 2013-11-04 13:05:48.354547 7f1cd6f5d820 0 ERROR: FCGX_Accept_r returned -88 > > Full output: http://pastebin.com/gyhQnrgP error 88 is "not a socket", which might hint that you didn't set 'rgw socket path' correctly. Yehuda ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Very frustrated with Ceph!
On 05/11/13 06:37, Alfredo Deza wrote: On Mon, Nov 4, 2013 at 12:25 PM, Gruher, Joseph R wrote: Could these problems be caused by running a purgedata but not a purge? It could be, I am not clear on what the expectation was for just doing purgedata without a purge. Purgedata removes /etc/ceph but without the purge ceph is still installed, then ceph-deploy install detects ceph as already installed and does not (re)create /etc/ceph? ceph-deploy will not create directories for you, that is left to the ceph install process, and just to be clear, the latest ceph-deploy version (1.3) does not remote /etc/ceph, just the contents. Yeah, however purgedata is removing /var/lib/ceph, which means after running purgedata you need to either run purge then install or manually recreate the various working directories under /var/lib/ceph before attempting any mon. mds or osd creation. Maybe purgedata should actually leave those top level dirs under /var/lib/ceph? regards Mark ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Very frustrated with Ceph!
Purgedata is only meant to be run *after* the package is uninstalled. We should make it do a check to enforce that. Otherwise we run into these problems... Mark Kirkwood wrote: >On 05/11/13 06:37, Alfredo Deza wrote: >> On Mon, Nov 4, 2013 at 12:25 PM, Gruher, Joseph R >> wrote: >>> Could these problems be caused by running a purgedata but not a >purge? >> >> It could be, I am not clear on what the expectation was for just >doing >> purgedata without a purge. >> >>> Purgedata removes /etc/ceph but without the purge ceph is still >installed, >>> then ceph-deploy install detects ceph as already installed and does >not >>> (re)create /etc/ceph? >> >> ceph-deploy will not create directories for you, that is left to the >> ceph install process, and just to be clear, the >> latest ceph-deploy version (1.3) does not remote /etc/ceph, just the >contents. > >Yeah, however purgedata is removing /var/lib/ceph, which means after >running purgedata you need to either run purge then install or manually > >recreate the various working directories under /var/lib/ceph before >attempting any mon. mds or osd creation. > >Maybe purgedata should actually leave those top level dirs under >/var/lib/ceph? > >regards > >Mark >___ >ceph-users mailing list >ceph-users@lists.ceph.com >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Multiple zones in different clusters
Hi list: 0.72 Emperor document shows us ,we can deploy two zones in a cluster to have a test .I tried to build two zones in different clusters , failed now! After some tests ,I find master zone can not be wroten in . Using the S3 API ,also I can not create a new bucket in the master zone ,though I can read info from the cluster. I'd like to know ,wheather the user just like "us-east" need to some special operation on its permission ,after its creation . Or maybe , with 0.72v ceph we can't just distribute two zones in different clusters! lixuehui___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] can not create bucket
| hi all! I configuration a ceph object gateway ! but I cant cleate bucket ! the ragosgw.log is rgw_create_bucket return ret=-95 bucket=mybucket(@.rgw.buckets[5510,25]) WARNING:set_req_state_err_err_no=95 resorting to 500 req 43:0.003689:s3:PUT /mybucket/:create_bucket:http_status=500 = req done req=0x143b110 http_status =500 == thinks peng | | | | | |___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] can not create bucket
error 95 is "Not supported", might mean that there are some issues with the osd, e.g., incompatibility (running older version than the gateway's), or likely objclass issues. What does the osd log say? Yehuda On Mon, Nov 4, 2013 at 6:50 PM, 鹏 wrote: > > hi all! > I configuration a ceph object gateway ! but I cant cleate bucket ! > > the ragosgw.log is > >rgw_create_bucket return ret=-95 bucket=mybucket(@.rgw.buckets[5510,25]) >WARNING:set_req_state_err_err_no=95 resorting to 500 >req 43:0.003689:s3:PUT /mybucket/:create_bucket:http_status=500 > = req done req=0x143b110 http_status =500 == > > > thinks > peng > > > > > > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Ceph::RADOS list_pools causes sefgault
Hello All,
I would like to control my Ceph cluster via Perl scripts. After a bit of
searching, I found a Perl module that was started back in 2011 [1].
It seems to work great except for the list_pools function which causes a
segfault.
We can observe this behavior when running a stacktrace on the
testrados2.plscript that comes with the attachment [2] (I added a
print "Testing
list_pools\n" before the actual call to $c->list_pools to make debugging
easier):
>> write(1, "Testing list_pools\n", 19Testing list_pools
>> )= 19
>> --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR,
si_addr=0x7fffd5dd5000} ---
>> +++ killed by SIGSEGV (core dumped) +++
>> Segmentation fault (core dumped)
Running this code through gdb I get possibly a little more helpful error:
( gdb ; set args testrados2.pl ; run )
>> Testing list_pools
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> __memset_sse2 () at ../sysdeps/x86_64/multiarch/../memset.S:913
>> 913 ../sysdeps/x86_64/multiarch/../memset.S: No such file or
directory.
But googling that error has landed me at a dead end.
I think this error is coming from the c function "list_pools_c" because of
the nature of the error.
I was hoping someone could help me debug the error and possibly point me in
a direction for extending librbd to manage images.
Thanks,
Jon A
[1] http://www.spinics.net/lists/ceph-devel/msg04147.html
[2] http://www.spinics.net/lists/ceph-devel/attachments/gzP0nHuUkx0f.gz
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] new version of stgt pushed to ceph-extras
Hey, all, we just refreshed stgt to its latest released version (1.0.41), and I also tweaked the rbd backend to be a little more flexible and useful. stgt is a userspace iSCSI target implementation (using tgtd) that can export several types of storage entities as iSCSI LUNs; backends include files in synchronous or asynchronous mode, "passthrough" to real SCSI devices, tape-device emulation on a file, sheepdog block images, and Ceph RBD images. New bs_rbd features include: * fixed up tgt-admin to work with rbd images (so .conf files work) * no more 20-rbd-image-per-tgtd limit * tgtadm accepts --bsopts for each image conf= to set path to ceph.conf file id= to set clientid This means that each image can have different logging and access rights, or even refer to a different cluster. The stgt source has also been refactored so that packagers can build with or without bs_rbd, now built into an .so, and distribute that module separately if desired; thus the base package doesn't require Ceph libraries librbd and librados. The source is available upstream at http://github.com/fujita/stgt. Packages are built and available in the ceph-extras repository at http://ceph.com/packages/ceph-extras/. Enjoy! -- Dan Mick, Filesystem Engineering Inktank Storage, Inc. http://inktank.com Ceph docs: http://ceph.com/docs ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] 0.72v ceph s3 API create bucket failed
Hi all
I tried to deploy a cluster with 0.72 ,the S3 api of ceph(0.72v) about user's
permission confused me .
this the user's info :
{ "user_id": "johndoe",
"display_name": "John Doe",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
{ "user": "johndoe",
"access_key": "PMC2FNSMNFC18B5360EC",
"secret_key": "m9\/D+Ei5XDY+uGAueYaluXfapgQicOTcUZiwlvo3"}],
"swift_keys": [],
"caps": [
{ "type": "usage",
"perm": "*"},
{ "type": "user",
"perm": "*"}],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": { "enabled": false,
"max_size_kb": -1,
"max_objects": -1}}
but when i using s3 api to create bucket with the user.
Traceback (most recent call last):
File "sss.py", line 13, in
conn.create_bucket('my_bucket')
File
"/usr/local/lib/python2.7/dist-packages/boto-2.13.3-py2.7.egg/boto/s3/connection.py",
line 504, in create_bucket
response.status, response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
AccessDenied
on the radosgw we get the info:
2013-11-05 15:14:17.281833 7fa14afa5700 1 == starting new request
req=0x2050f40 =
2013-11-05 15:14:17.284046 7fa14afa5700 1 == req done req=0x2050f40
http_status=403 ==
the .conf file about the gateway is :
host = ceph-rgw40
keyring = /etc/ceph/keyring.radosgw.gateway
rgw socket path = /tmp/radosgw.sock
log file = /var/log/ceph/radosgw.log
rgw dns name =ceph-osd26
rgw print continue =false
the rgw.conf is :
FastCgiExternalServer /var/www/s3gw.fcgi -socket /tmp/radosgw.sock
ServerName cephcom.com
#
#
#ServerAdmin {email.address}
DocumentRoot /var/www
RewriteEngine On
RewriteRule ^/(.*)
/s3gw.fcgi?%{QUERY_STRING}[E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
Options +ExecCGI
AllowOverride All
SetHandler fastcgi-script
Order allow,deny
Allow from all
AuthBasicAuthoritative Off
AllowEncodedSlashes On
ErrorLog /var/log/apache2/error.log
CustomLog /var/log/apache2/access.log combined
ServerSignature Off
and dns server seems ok ,when i ping the domain name "cephcom.com":
ping cephcom.com
PING cephcom.com (192.168.50.92) 56(84) bytes of data.
64 bytes from ceph-rgw40 (192.168.50.92): icmp_req=1 ttl=64 time=0.115 ms
64 bytes from ceph-rgw40 (192.168.50.92): icmp_req=2 ttl=64 time=0.084 ms
thank for any help !
lixuehui___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
