[ceph-users] what are these files for mon?
Hello, I’ve been testing cephfs with 1 monitor. My /var partition keeps on filling up so that the mon process just die because of insufficient space. I drilled down on /var partition that below mon path is taking most of the space with *.sst files. I just curious what these files are and can they be archived or removed? [root@gfsnode5 store.db]# pwd /var/lib/ceph/mon/ceph-gfsnode5/store.db [root@gfsnode5 store.db]# ls *.sst | wc -l 1816 [root@gfsnode5 store.db]# Thanks, Jimmy ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] OSD keyrings shifted and down
More detail to this. I recently upgraded my Ceph cluster from Emperor to Firefly. After the upgrade had been done, I noticed 1 of the OSD not coming back to life. While in the process of troubleshooting, rebooted the osd server and the keyring shifted. My $ENV. 4x OSD servers (each has 12, 1 for root and 11 for OSD) 1x mon + mds + admin for ceph-deploy Hopefully someone out there experience similar situation, if you do, please share your fixes. Thanks, Jimmy From: J L j...@yahoo-inc.commailto:j...@yahoo-inc.com Date: Friday, June 6, 2014 at 1:13 PM To: J L j...@yahoo-inc.commailto:j...@yahoo-inc.com, ceph-users@lists.ceph.commailto:ceph-users@lists.ceph.com ceph-users@lists.ceph.commailto:ceph-users@lists.ceph.com Subject: Re: [ceph-users] OSD keyrings shifted and down Has anyone run into this issue and would like to provide any troubleshooting tip? Thanks, Jimmy From: J L j...@yahoo-inc.commailto:j...@yahoo-inc.com Date: Thursday, June 5, 2014 at 4:20 PM To: ceph-users@lists.ceph.commailto:ceph-users@lists.ceph.com ceph-users@lists.ceph.commailto:ceph-users@lists.ceph.com Subject: [ceph-users] OSD keyrings shifted and down Hello Ceph Guru, I rebooted osd server to fix “osd.33”. When the server came back online, I noticed all the osd are down, while I am troubleshooting and restarting the osd, I got below error for authentication. I also noticed the “keyring” for each osd had shifted. For example, for osd.33 which mapped to /var/lib/ceph/osd/ceph-33, its keyring should be mapped to [osd.33], in this case it mapped to [osd.34]. Can I just simply change the osd.# in the keyring to correct the mapping or is there proper for the fix? Please help. Thanks in advance!! -Jimmy [root@gfsnode1 ceph-34]# service ceph start osd.34 === osd.34 === 2014-06-05 15:08:54.053958 7f08f2b47700 0 librados: osd.34 authentication error (1) Operation not permitted Error connecting to cluster: PermissionError failed: 'timeout 30 /usr/bin/ceph -c /etc/ceph/ceph.conf --name=osd.34 --keyring=/var/lib/ceph/osd/ceph-34/keyring osd crush create-or-move -- 34 2.73 host=gfsnode1 root=default' [root@gfsnode1 ceph-34]# [root@gfsnode1 osd]# ls -l total 0 lrwxrwxrwx 1 root root 12 May 15 15:21 ceph-33 - /ceph/osd120 lrwxrwxrwx 1 root root 12 May 15 15:22 ceph-34 - /ceph/osd121 lrwxrwxrwx 1 root root 12 May 15 15:23 ceph-35 - /ceph/osd122 lrwxrwxrwx 1 root root 12 May 15 15:24 ceph-36 - /ceph/osd123 lrwxrwxrwx 1 root root 12 May 15 15:24 ceph-37 - /ceph/osd124 lrwxrwxrwx 1 root root 12 May 15 15:25 ceph-38 - /ceph/osd125 lrwxrwxrwx 1 root root 12 May 15 15:25 ceph-39 - /ceph/osd126 lrwxrwxrwx 1 root root 12 May 15 15:26 ceph-40 - /ceph/osd127 lrwxrwxrwx 1 root root 12 May 15 15:27 ceph-41 - /ceph/osd128 lrwxrwxrwx 1 root root 12 May 15 15:27 ceph-42 - /ceph/osd129 lrwxrwxrwx 1 root root 12 May 15 15:28 ceph-43 - /ceph/osd130 [root@gfsnode1 osd]# cat ceph-33/keyring [osd.34] key = AQAwPnVT6G7fBRAA86D4FuxN0U8uKXk0brPbCQ== [root@gfsnode1 osd]# cat ceph-34/keyring [osd.35] key = AQBbPnVTmG4BLxAA6UV6XHbZepXUEXB6VJQzEA== [root@gfsnode1 osd]# cat ceph-35/keyring [osd.36] key = AQCDPnVTuL97JRAA1soDHToJ1c6WhXX+mnnRPw== [root@gfsnode1 osd]# cat ceph-36/keyring [osd.37] key = AQCwPnVTYAttNhAAomeRalOEHWlyO7C9tF+7SQ== [root@gfsnode1 osd]# cat ceph-37/keyring [osd.38] key = AQDKPnVTQC1DLBAAl0959S0st+UcFw8uOppa7g== [root@gfsnode1 osd]# cat ceph-38/keyring [osd.39] key = AQDjPnVTMFGwNxAABH5M1Y8uXoqecPesS09IGw== [root@gfsnode1 osd]# cat ceph-39/keyring [osd.40] key = AQChQXVT6JHiBxAAohTnBGxb2ZAbgCjt5M0xBw== [root@gfsnode1 osd]# cat ceph-40/keyring [osd.41] key = AQBGP3VTAHI0CRAAZkcUPLOFT1jx9v3DVNX4nQ== [root@gfsnode1 osd]# cat ceph-41/keyring [osd.42] key = AQAEsIdTMBTjChAAfJrsqIEBcCGEXv0jcK2vtQ== [root@gfsnode1 osd]# cat ceph-42/keyring [osd.43] key = AQB6P3VT2KW7ORAAU+1Ix/fUXIBU8jky0BQ9jw== [root@gfsnode1 osd]# cat ceph-43/keyring cat: ceph-43/keyring: No such file or directory [root@gfsnode1 osd]# ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] OSD keyrings shifted and down
Has anyone run into this issue and would like to provide any troubleshooting tip? Thanks, Jimmy From: J L j...@yahoo-inc.commailto:j...@yahoo-inc.com Date: Thursday, June 5, 2014 at 4:20 PM To: ceph-users@lists.ceph.commailto:ceph-users@lists.ceph.com ceph-users@lists.ceph.commailto:ceph-users@lists.ceph.com Subject: [ceph-users] OSD keyrings shifted and down Hello Ceph Guru, I rebooted osd server to fix “osd.33”. When the server came back online, I noticed all the osd are down, while I am troubleshooting and restarting the osd, I got below error for authentication. I also noticed the “keyring” for each osd had shifted. For example, for osd.33 which mapped to /var/lib/ceph/osd/ceph-33, its keyring should be mapped to [osd.33], in this case it mapped to [osd.34]. Can I just simply change the osd.# in the keyring to correct the mapping or is there proper for the fix? Please help. Thanks in advance!! -Jimmy [root@gfsnode1 ceph-34]# service ceph start osd.34 === osd.34 === 2014-06-05 15:08:54.053958 7f08f2b47700 0 librados: osd.34 authentication error (1) Operation not permitted Error connecting to cluster: PermissionError failed: 'timeout 30 /usr/bin/ceph -c /etc/ceph/ceph.conf --name=osd.34 --keyring=/var/lib/ceph/osd/ceph-34/keyring osd crush create-or-move -- 34 2.73 host=gfsnode1 root=default' [root@gfsnode1 ceph-34]# [root@gfsnode1 osd]# ls -l total 0 lrwxrwxrwx 1 root root 12 May 15 15:21 ceph-33 - /ceph/osd120 lrwxrwxrwx 1 root root 12 May 15 15:22 ceph-34 - /ceph/osd121 lrwxrwxrwx 1 root root 12 May 15 15:23 ceph-35 - /ceph/osd122 lrwxrwxrwx 1 root root 12 May 15 15:24 ceph-36 - /ceph/osd123 lrwxrwxrwx 1 root root 12 May 15 15:24 ceph-37 - /ceph/osd124 lrwxrwxrwx 1 root root 12 May 15 15:25 ceph-38 - /ceph/osd125 lrwxrwxrwx 1 root root 12 May 15 15:25 ceph-39 - /ceph/osd126 lrwxrwxrwx 1 root root 12 May 15 15:26 ceph-40 - /ceph/osd127 lrwxrwxrwx 1 root root 12 May 15 15:27 ceph-41 - /ceph/osd128 lrwxrwxrwx 1 root root 12 May 15 15:27 ceph-42 - /ceph/osd129 lrwxrwxrwx 1 root root 12 May 15 15:28 ceph-43 - /ceph/osd130 [root@gfsnode1 osd]# cat ceph-33/keyring [osd.34] key = AQAwPnVT6G7fBRAA86D4FuxN0U8uKXk0brPbCQ== [root@gfsnode1 osd]# cat ceph-34/keyring [osd.35] key = AQBbPnVTmG4BLxAA6UV6XHbZepXUEXB6VJQzEA== [root@gfsnode1 osd]# cat ceph-35/keyring [osd.36] key = AQCDPnVTuL97JRAA1soDHToJ1c6WhXX+mnnRPw== [root@gfsnode1 osd]# cat ceph-36/keyring [osd.37] key = AQCwPnVTYAttNhAAomeRalOEHWlyO7C9tF+7SQ== [root@gfsnode1 osd]# cat ceph-37/keyring [osd.38] key = AQDKPnVTQC1DLBAAl0959S0st+UcFw8uOppa7g== [root@gfsnode1 osd]# cat ceph-38/keyring [osd.39] key = AQDjPnVTMFGwNxAABH5M1Y8uXoqecPesS09IGw== [root@gfsnode1 osd]# cat ceph-39/keyring [osd.40] key = AQChQXVT6JHiBxAAohTnBGxb2ZAbgCjt5M0xBw== [root@gfsnode1 osd]# cat ceph-40/keyring [osd.41] key = AQBGP3VTAHI0CRAAZkcUPLOFT1jx9v3DVNX4nQ== [root@gfsnode1 osd]# cat ceph-41/keyring [osd.42] key = AQAEsIdTMBTjChAAfJrsqIEBcCGEXv0jcK2vtQ== [root@gfsnode1 osd]# cat ceph-42/keyring [osd.43] key = AQB6P3VT2KW7ORAAU+1Ix/fUXIBU8jky0BQ9jw== [root@gfsnode1 osd]# cat ceph-43/keyring cat: ceph-43/keyring: No such file or directory [root@gfsnode1 osd]# ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] OSD keyrings shifted and down
Hello Ceph Guru, I rebooted osd server to fix “osd.33”. When the server came back online, I noticed all the osd are down, while I am troubleshooting and restarting the osd, I got below error for authentication. I also noticed the “keyring” for each osd had shifted. For example, for osd.33 which mapped to /var/lib/ceph/osd/ceph-33, its keyring should be mapped to [osd.33], in this case it mapped to [osd.34]. Can I just simply change the osd.# in the keyring to correct the mapping or is there proper for the fix? Please help. Thanks in advance!! -Jimmy [root@gfsnode1 ceph-34]# service ceph start osd.34 === osd.34 === 2014-06-05 15:08:54.053958 7f08f2b47700 0 librados: osd.34 authentication error (1) Operation not permitted Error connecting to cluster: PermissionError failed: 'timeout 30 /usr/bin/ceph -c /etc/ceph/ceph.conf --name=osd.34 --keyring=/var/lib/ceph/osd/ceph-34/keyring osd crush create-or-move -- 34 2.73 host=gfsnode1 root=default' [root@gfsnode1 ceph-34]# [root@gfsnode1 osd]# ls -l total 0 lrwxrwxrwx 1 root root 12 May 15 15:21 ceph-33 - /ceph/osd120 lrwxrwxrwx 1 root root 12 May 15 15:22 ceph-34 - /ceph/osd121 lrwxrwxrwx 1 root root 12 May 15 15:23 ceph-35 - /ceph/osd122 lrwxrwxrwx 1 root root 12 May 15 15:24 ceph-36 - /ceph/osd123 lrwxrwxrwx 1 root root 12 May 15 15:24 ceph-37 - /ceph/osd124 lrwxrwxrwx 1 root root 12 May 15 15:25 ceph-38 - /ceph/osd125 lrwxrwxrwx 1 root root 12 May 15 15:25 ceph-39 - /ceph/osd126 lrwxrwxrwx 1 root root 12 May 15 15:26 ceph-40 - /ceph/osd127 lrwxrwxrwx 1 root root 12 May 15 15:27 ceph-41 - /ceph/osd128 lrwxrwxrwx 1 root root 12 May 15 15:27 ceph-42 - /ceph/osd129 lrwxrwxrwx 1 root root 12 May 15 15:28 ceph-43 - /ceph/osd130 [root@gfsnode1 osd]# cat ceph-33/keyring [osd.34] key = AQAwPnVT6G7fBRAA86D4FuxN0U8uKXk0brPbCQ== [root@gfsnode1 osd]# cat ceph-34/keyring [osd.35] key = AQBbPnVTmG4BLxAA6UV6XHbZepXUEXB6VJQzEA== [root@gfsnode1 osd]# cat ceph-35/keyring [osd.36] key = AQCDPnVTuL97JRAA1soDHToJ1c6WhXX+mnnRPw== [root@gfsnode1 osd]# cat ceph-36/keyring [osd.37] key = AQCwPnVTYAttNhAAomeRalOEHWlyO7C9tF+7SQ== [root@gfsnode1 osd]# cat ceph-37/keyring [osd.38] key = AQDKPnVTQC1DLBAAl0959S0st+UcFw8uOppa7g== [root@gfsnode1 osd]# cat ceph-38/keyring [osd.39] key = AQDjPnVTMFGwNxAABH5M1Y8uXoqecPesS09IGw== [root@gfsnode1 osd]# cat ceph-39/keyring [osd.40] key = AQChQXVT6JHiBxAAohTnBGxb2ZAbgCjt5M0xBw== [root@gfsnode1 osd]# cat ceph-40/keyring [osd.41] key = AQBGP3VTAHI0CRAAZkcUPLOFT1jx9v3DVNX4nQ== [root@gfsnode1 osd]# cat ceph-41/keyring [osd.42] key = AQAEsIdTMBTjChAAfJrsqIEBcCGEXv0jcK2vtQ== [root@gfsnode1 osd]# cat ceph-42/keyring [osd.43] key = AQB6P3VT2KW7ORAAU+1Ix/fUXIBU8jky0BQ9jw== [root@gfsnode1 osd]# cat ceph-43/keyring cat: ceph-43/keyring: No such file or directory [root@gfsnode1 osd]# ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to find the disk partitions attached to a OSD
This would give you pretty good understanding where the mounts and /dev/sd* are. [jlu@gfsnode1 osd]$ ceph-disk list; pwd; ls -lai /dev/sda : /dev/sda1 other, mounted on /boot /dev/sda2 other /dev/sdb other, unknown, mounted on /ceph/osd120 /dev/sdc other, unknown, mounted on /ceph/osd121 /dev/sdd other, unknown, mounted on /ceph/osd122 /dev/sde other, unknown, mounted on /ceph/osd123 /dev/sdf other, unknown, mounted on /ceph/osd124 /dev/sdg other, unknown, mounted on /ceph/osd125 /dev/sdh other, unknown, mounted on /ceph/osd126 /dev/sdi other, unknown, mounted on /ceph/osd127 /dev/sdj other, unknown, mounted on /ceph/osd128 /dev/sdk other, unknown, mounted on /ceph/osd129 /dev/sdl other, unknown, mounted on /ceph/osd130 /var/lib/ceph/osd total 8 497 drwxr-xr-x 2 root root 4096 May 15 15:28 . 492 drwxr-xr-x 8 root root 4096 May 15 15:13 .. 725 lrwxrwxrwx 1 root root 12 May 15 15:21 ceph-33 - /ceph/osd120 782 lrwxrwxrwx 1 root root 12 May 15 15:22 ceph-34 - /ceph/osd121 1081 lrwxrwxrwx 1 root root 12 May 15 15:23 ceph-35 - /ceph/osd122 1095 lrwxrwxrwx 1 root root 12 May 15 15:24 ceph-36 - /ceph/osd123 1129 lrwxrwxrwx 1 root root 12 May 15 15:24 ceph-37 - /ceph/osd124 1137 lrwxrwxrwx 1 root root 12 May 15 15:25 ceph-38 - /ceph/osd125 1143 lrwxrwxrwx 1 root root 12 May 15 15:25 ceph-39 - /ceph/osd126 1185 lrwxrwxrwx 1 root root 12 May 15 15:26 ceph-40 - /ceph/osd127 1242 lrwxrwxrwx 1 root root 12 May 15 15:27 ceph-41 - /ceph/osd128 1398 lrwxrwxrwx 1 root root 12 May 15 15:27 ceph-42 - /ceph/osd129 1437 lrwxrwxrwx 1 root root 12 May 15 15:28 ceph-43 - /ceph/osd130 [jlu@gfsnode1 osd]$ -Jimmy On 5/21/14, 8:33 AM, Sage Weil s...@inktank.com wrote: You might also try ceph-disk list sage On Wed, 21 May 2014, Mike Dawson wrote: Looks like you may not have any OSDs properly setup and mounted. It should look more like: user@host:~# mount | grep ceph /dev/sdb1 on /var/lib/ceph/osd/ceph-0 type xfs (rw,noatime,inode64) /dev/sdc1 on /var/lib/ceph/osd/ceph-1 type xfs (rw,noatime,inode64) /dev/sdd1 on /var/lib/ceph/osd/ceph-2 type xfs (rw,noatime,inode64) Confirm the OSD in your ceph cluster with: user@host:~# ceph osd tree - Mike On 5/21/2014 11:15 AM, Sharmila Govind wrote: Hi Mike, Thanks for your quick response. When I try mount on the storage node this is what I get: *root@cephnode4:~# mount* */dev/sda1 on / type ext4 (rw,errors=remount-ro)* *proc on /proc type proc (rw,noexec,nosuid,nodev)* *sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)* *none on /sys/fs/fuse/connections type fusectl (rw)* *none on /sys/kernel/debug type debugfs (rw)* *none on /sys/kernel/security type securityfs (rw)* *udev on /dev type devtmpfs (rw,mode=0755)* *devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)* *tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)* *none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)* *none on /run/shm type tmpfs (rw,nosuid,nodev)* */dev/sdb on /mnt/CephStorage1 type ext4 (rw)* */dev/sdc on /mnt/CephStorage2 type ext4 (rw)* */dev/sda7 on /mnt/Storage type ext4 (rw)* */dev/sda2 on /boot type ext4 (rw)* */dev/sda5 on /home type ext4 (rw)* */dev/sda6 on /mnt/CephStorage type ext4 (rw)* Is there anything wrong in the setup I have? I dont have any 'ceph' related mounts. Thanks, Sharmila On Wed, May 21, 2014 at 8:34 PM, Mike Dawson mike.daw...@cloudapt.com mailto:mike.daw...@cloudapt.com wrote: Perhaps: # mount | grep ceph - Mike Dawson On 5/21/2014 11:00 AM, Sharmila Govind wrote: Hi, I am new to Ceph. I have a storage node with 2 OSDs. Iam trying to figure out to which pyhsical device/partition each of the OSDs are attached to. Is there are command that can be executed in the storage node to find out the same. Thanks in Advance, Sharmila _ ceph-users mailing list ceph-users@lists.ceph.com mailto:ceph-users@lists.ceph.com http://lists.ceph.com/__listinfo.cgi/ceph-users-ceph.__com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] ceph mom help
Hello All,
I am using “ceph-deploy” to setup my ceph cluster with 3 nodes. I am getting an
error when running “sudo ceph-deploy mon create gfsnode5”. Would you someone
please give me a pointer what the problem is?
Thanks in advance!
-Jimmy
[cuser@gfsnode5 my-cluster]$ sudo ceph-deploy mon create gfsnode5
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.1): /usr/bin/ceph-deploy mon create
gfsnode5
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts gfsnode5
[ceph_deploy.mon][DEBUG ] detecting platform for host gfsnode5 ...
[gfsnode5][DEBUG ] connected to host: gfsnode5
[gfsnode5][DEBUG ] detect platform information from remote host
[gfsnode5][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: Red Hat Enterprise Linux Server 6.5
Santiago
[gfsnode5][DEBUG ] determining if provided host has same hostname in remote
[gfsnode5][DEBUG ] get remote short hostname
[gfsnode5][DEBUG ] deploying mon to gfsnode5
[gfsnode5][DEBUG ] get remote short hostname
[gfsnode5][DEBUG ] remote hostname: gfsnode5
[gfsnode5][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[gfsnode5][DEBUG ] create the mon path if it does not exist
[gfsnode5][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-gfsnode5/done
[gfsnode5][DEBUG ] done path does not exist:
/var/lib/ceph/mon/ceph-gfsnode5/done
[gfsnode5][INFO ] creating keyring file:
/var/lib/ceph/tmp/ceph-gfsnode5.mon.keyring
[gfsnode5][DEBUG ] create the monitor keyring file
[gfsnode5][INFO ] Running command: sudo ceph-mon --cluster ceph --mkfs -i
gfsnode5 --keyring /var/lib/ceph/tmp/ceph-gfsnode5.mon.keyring
[gfsnode5][WARNIN] pthread lock: Invalid argument
[gfsnode5][WARNIN] *** Caught signal (Aborted) **
[gfsnode5][WARNIN] in thread 7f22e8d737a0
[gfsnode5][WARNIN] ceph version 0.72.2
(a913ded2ff138aefb8cb84d347d72164099cfd60)
[gfsnode5][WARNIN] 1: ceph-mon() [0x808571]
[gfsnode5][WARNIN] 2: /lib64/libpthread.so.0() [0x34aec0f710]
[gfsnode5][WARNIN] 3: (gsignal()+0x35) [0x34ae832925]
[gfsnode5][WARNIN] 4: (abort()+0x175) [0x34ae834105]
[gfsnode5][WARNIN] 5: /usr/lib64/libleveldb.so.1() [0x33bf034d71]
[gfsnode5][WARNIN] 6: (leveldb::DBImpl::Get(leveldb::ReadOptions const,
leveldb::Slice const, leveldb::Value*)+0x50) [0x33bf01a120]
[gfsnode5][WARNIN] 7: (LevelDBStore::_get_iterator()+0x41) [0x7773b1]
[gfsnode5][WARNIN] 8: (MonitorDBStore::exists(std::string const, std::string
const)+0x28) [0x5276f8]
[gfsnode5][WARNIN] 9: (Monitor::check_fsid()+0x73) [0x550bb3]
[gfsnode5][WARNIN] 10: (Monitor::mkfs(ceph::buffer::list)+0x47) [0x55f027]
[gfsnode5][WARNIN] 11: (main()+0x11d7) [0x521f97]
[gfsnode5][WARNIN] 12: (__libc_start_main()+0xfd) [0x34ae81ed1d]
[gfsnode5][WARNIN] 13: ceph-mon() [0x51fa79]
[gfsnode5][WARNIN] 2014-05-02 11:19:46.335854 7f22e8d737a0 -1 *** Caught signal
(Aborted) **
[gfsnode5][WARNIN] in thread 7f22e8d737a0
[gfsnode5][WARNIN]
[gfsnode5][WARNIN] ceph version 0.72.2
(a913ded2ff138aefb8cb84d347d72164099cfd60)
[gfsnode5][WARNIN] 1: ceph-mon() [0x808571]
[gfsnode5][WARNIN] 2: /lib64/libpthread.so.0() [0x34aec0f710]
[gfsnode5][WARNIN] 3: (gsignal()+0x35) [0x34ae832925]
[gfsnode5][WARNIN] 4: (abort()+0x175) [0x34ae834105]
[gfsnode5][WARNIN] 5: /usr/lib64/libleveldb.so.1() [0x33bf034d71]
[gfsnode5][WARNIN] 6: (leveldb::DBImpl::Get(leveldb::ReadOptions const,
leveldb::Slice const, leveldb::Value*)+0x50) [0x33bf01a120]
[gfsnode5][WARNIN] 7: (LevelDBStore::_get_iterator()+0x41) [0x7773b1]
[gfsnode5][WARNIN] 8: (MonitorDBStore::exists(std::string const, std::string
const)+0x28) [0x5276f8]
[gfsnode5][WARNIN] 9: (Monitor::check_fsid()+0x73) [0x550bb3]
[gfsnode5][WARNIN] 10: (Monitor::mkfs(ceph::buffer::list)+0x47) [0x55f027]
[gfsnode5][WARNIN] 11: (main()+0x11d7) [0x521f97]
[gfsnode5][WARNIN] 12: (__libc_start_main()+0xfd) [0x34ae81ed1d]
[gfsnode5][WARNIN] 13: ceph-mon() [0x51fa79]
[gfsnode5][WARNIN] NOTE: a copy of the executable, or `objdump -rdS
executable` is needed to interpret this.
[gfsnode5][WARNIN]
[gfsnode5][WARNIN] 0 2014-05-02 11:19:46.335854 7f22e8d737a0 -1 ***
Caught signal (Aborted) **
[gfsnode5][WARNIN] in thread 7f22e8d737a0
[gfsnode5][WARNIN]
[gfsnode5][WARNIN] ceph version 0.72.2
(a913ded2ff138aefb8cb84d347d72164099cfd60)
[gfsnode5][WARNIN] 1: ceph-mon() [0x808571]
[gfsnode5][WARNIN] 2: /lib64/libpthread.so.0() [0x34aec0f710]
[gfsnode5][WARNIN] 3: (gsignal()+0x35) [0x34ae832925]
[gfsnode5][WARNIN] 4: (abort()+0x175) [0x34ae834105]
[gfsnode5][WARNIN] 5: /usr/lib64/libleveldb.so.1() [0x33bf034d71]
[gfsnode5][WARNIN] 6: (leveldb::DBImpl::Get(leveldb::ReadOptions const,
leveldb::Slice const, leveldb::Value*)+0x50) [0x33bf01a120]
[gfsnode5][WARNIN] 7: (LevelDBStore::_get_iterator()+0x41) [0x7773b1]
[gfsnode5][WARNIN] 8: (MonitorDBStore::exists(std::string const, std::string
const)+0x28) [0x5276f8]
[gfsnode5][WARNIN] 9:
Re: [ceph-users] ceph mom help
This is a possible bug which was resolved. It was due to leveldb version. My
node is already running on version 1.12.
[root@gfsnode5 my-cluster]# rpm -qa | grep -i leveldb
leveldb-1.12.0-3.el6.x86_64
[root@gfsnode5 my-cluster]#
Thanks,
Jimmy
From: J L j...@yahoo-inc.commailto:j...@yahoo-inc.com
Date: Friday, May 2, 2014 at 11:21 AM
To: ceph-us...@ceph.commailto:ceph-us...@ceph.com
ceph-us...@ceph.commailto:ceph-us...@ceph.com
Subject: ceph mom help
Hello All,
I am using “ceph-deploy” to setup my ceph cluster with 3 nodes. I am getting an
error when running “sudo ceph-deploy mon create gfsnode5”. Would you someone
please give me a pointer what the problem is?
Thanks in advance!
-Jimmy
[cuser@gfsnode5 my-cluster]$ sudo ceph-deploy mon create gfsnode5
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.1): /usr/bin/ceph-deploy mon create
gfsnode5
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts gfsnode5
[ceph_deploy.mon][DEBUG ] detecting platform for host gfsnode5 ...
[gfsnode5][DEBUG ] connected to host: gfsnode5
[gfsnode5][DEBUG ] detect platform information from remote host
[gfsnode5][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: Red Hat Enterprise Linux Server 6.5
Santiago
[gfsnode5][DEBUG ] determining if provided host has same hostname in remote
[gfsnode5][DEBUG ] get remote short hostname
[gfsnode5][DEBUG ] deploying mon to gfsnode5
[gfsnode5][DEBUG ] get remote short hostname
[gfsnode5][DEBUG ] remote hostname: gfsnode5
[gfsnode5][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[gfsnode5][DEBUG ] create the mon path if it does not exist
[gfsnode5][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-gfsnode5/done
[gfsnode5][DEBUG ] done path does not exist:
/var/lib/ceph/mon/ceph-gfsnode5/done
[gfsnode5][INFO ] creating keyring file:
/var/lib/ceph/tmp/ceph-gfsnode5.mon.keyring
[gfsnode5][DEBUG ] create the monitor keyring file
[gfsnode5][INFO ] Running command: sudo ceph-mon --cluster ceph --mkfs -i
gfsnode5 --keyring /var/lib/ceph/tmp/ceph-gfsnode5.mon.keyring
[gfsnode5][WARNIN] pthread lock: Invalid argument
[gfsnode5][WARNIN] *** Caught signal (Aborted) **
[gfsnode5][WARNIN] in thread 7f22e8d737a0
[gfsnode5][WARNIN] ceph version 0.72.2
(a913ded2ff138aefb8cb84d347d72164099cfd60)
[gfsnode5][WARNIN] 1: ceph-mon() [0x808571]
[gfsnode5][WARNIN] 2: /lib64/libpthread.so.0() [0x34aec0f710]
[gfsnode5][WARNIN] 3: (gsignal()+0x35) [0x34ae832925]
[gfsnode5][WARNIN] 4: (abort()+0x175) [0x34ae834105]
[gfsnode5][WARNIN] 5: /usr/lib64/libleveldb.so.1() [0x33bf034d71]
[gfsnode5][WARNIN] 6: (leveldb::DBImpl::Get(leveldb::ReadOptions const,
leveldb::Slice const, leveldb::Value*)+0x50) [0x33bf01a120]
[gfsnode5][WARNIN] 7: (LevelDBStore::_get_iterator()+0x41) [0x7773b1]
[gfsnode5][WARNIN] 8: (MonitorDBStore::exists(std::string const, std::string
const)+0x28) [0x5276f8]
[gfsnode5][WARNIN] 9: (Monitor::check_fsid()+0x73) [0x550bb3]
[gfsnode5][WARNIN] 10: (Monitor::mkfs(ceph::buffer::list)+0x47) [0x55f027]
[gfsnode5][WARNIN] 11: (main()+0x11d7) [0x521f97]
[gfsnode5][WARNIN] 12: (__libc_start_main()+0xfd) [0x34ae81ed1d]
[gfsnode5][WARNIN] 13: ceph-mon() [0x51fa79]
[gfsnode5][WARNIN] 2014-05-02 11:19:46.335854 7f22e8d737a0 -1 *** Caught signal
(Aborted) **
[gfsnode5][WARNIN] in thread 7f22e8d737a0
[gfsnode5][WARNIN]
[gfsnode5][WARNIN] ceph version 0.72.2
(a913ded2ff138aefb8cb84d347d72164099cfd60)
[gfsnode5][WARNIN] 1: ceph-mon() [0x808571]
[gfsnode5][WARNIN] 2: /lib64/libpthread.so.0() [0x34aec0f710]
[gfsnode5][WARNIN] 3: (gsignal()+0x35) [0x34ae832925]
[gfsnode5][WARNIN] 4: (abort()+0x175) [0x34ae834105]
[gfsnode5][WARNIN] 5: /usr/lib64/libleveldb.so.1() [0x33bf034d71]
[gfsnode5][WARNIN] 6: (leveldb::DBImpl::Get(leveldb::ReadOptions const,
leveldb::Slice const, leveldb::Value*)+0x50) [0x33bf01a120]
[gfsnode5][WARNIN] 7: (LevelDBStore::_get_iterator()+0x41) [0x7773b1]
[gfsnode5][WARNIN] 8: (MonitorDBStore::exists(std::string const, std::string
const)+0x28) [0x5276f8]
[gfsnode5][WARNIN] 9: (Monitor::check_fsid()+0x73) [0x550bb3]
[gfsnode5][WARNIN] 10: (Monitor::mkfs(ceph::buffer::list)+0x47) [0x55f027]
[gfsnode5][WARNIN] 11: (main()+0x11d7) [0x521f97]
[gfsnode5][WARNIN] 12: (__libc_start_main()+0xfd) [0x34ae81ed1d]
[gfsnode5][WARNIN] 13: ceph-mon() [0x51fa79]
[gfsnode5][WARNIN] NOTE: a copy of the executable, or `objdump -rdS
executable` is needed to interpret this.
[gfsnode5][WARNIN]
[gfsnode5][WARNIN] 0 2014-05-02 11:19:46.335854 7f22e8d737a0 -1 ***
Caught signal (Aborted) **
[gfsnode5][WARNIN] in thread 7f22e8d737a0
[gfsnode5][WARNIN]
[gfsnode5][WARNIN] ceph version 0.72.2
(a913ded2ff138aefb8cb84d347d72164099cfd60)
[gfsnode5][WARNIN] 1: ceph-mon() [0x808571]
[gfsnode5][WARNIN] 2: /lib64/libpthread.so.0() [0x34aec0f710]
[gfsnode5][WARNIN] 3: (gsignal()+0x35) [0x34ae832925]
Re: [ceph-users] ceph mom help
I was able to dig up an archive of an IRC chat from Sage. The suggestion from
the chat was to downgrade leveldb from 1.12 to 1.7.0. After the downgrade, I
was able to run sudo ceph-deploy mon create gfsnode5.
-Jimmy
From: J L j...@yahoo-inc.commailto:j...@yahoo-inc.com
Date: Friday, May 2, 2014 at 11:30 AM
To: ceph-us...@ceph.commailto:ceph-us...@ceph.com
ceph-us...@ceph.commailto:ceph-us...@ceph.com
Subject: Re: ceph mom help
This is a possible bug which was resolved. It was due to leveldb version. My
node is already running on version 1.12.
[root@gfsnode5 my-cluster]# rpm -qa | grep -i leveldb
leveldb-1.12.0-3.el6.x86_64
[root@gfsnode5 my-cluster]#
Thanks,
Jimmy
From: J L j...@yahoo-inc.commailto:j...@yahoo-inc.com
Date: Friday, May 2, 2014 at 11:21 AM
To: ceph-us...@ceph.commailto:ceph-us...@ceph.com
ceph-us...@ceph.commailto:ceph-us...@ceph.com
Subject: ceph mom help
Hello All,
I am using “ceph-deploy” to setup my ceph cluster with 3 nodes. I am getting an
error when running “sudo ceph-deploy mon create gfsnode5”. Would you someone
please give me a pointer what the problem is?
Thanks in advance!
-Jimmy
[cuser@gfsnode5 my-cluster]$ sudo ceph-deploy mon create gfsnode5
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.1): /usr/bin/ceph-deploy mon create
gfsnode5
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts gfsnode5
[ceph_deploy.mon][DEBUG ] detecting platform for host gfsnode5 ...
[gfsnode5][DEBUG ] connected to host: gfsnode5
[gfsnode5][DEBUG ] detect platform information from remote host
[gfsnode5][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: Red Hat Enterprise Linux Server 6.5
Santiago
[gfsnode5][DEBUG ] determining if provided host has same hostname in remote
[gfsnode5][DEBUG ] get remote short hostname
[gfsnode5][DEBUG ] deploying mon to gfsnode5
[gfsnode5][DEBUG ] get remote short hostname
[gfsnode5][DEBUG ] remote hostname: gfsnode5
[gfsnode5][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[gfsnode5][DEBUG ] create the mon path if it does not exist
[gfsnode5][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-gfsnode5/done
[gfsnode5][DEBUG ] done path does not exist:
/var/lib/ceph/mon/ceph-gfsnode5/done
[gfsnode5][INFO ] creating keyring file:
/var/lib/ceph/tmp/ceph-gfsnode5.mon.keyring
[gfsnode5][DEBUG ] create the monitor keyring file
[gfsnode5][INFO ] Running command: sudo ceph-mon --cluster ceph --mkfs -i
gfsnode5 --keyring /var/lib/ceph/tmp/ceph-gfsnode5.mon.keyring
[gfsnode5][WARNIN] pthread lock: Invalid argument
[gfsnode5][WARNIN] *** Caught signal (Aborted) **
[gfsnode5][WARNIN] in thread 7f22e8d737a0
[gfsnode5][WARNIN] ceph version 0.72.2
(a913ded2ff138aefb8cb84d347d72164099cfd60)
[gfsnode5][WARNIN] 1: ceph-mon() [0x808571]
[gfsnode5][WARNIN] 2: /lib64/libpthread.so.0() [0x34aec0f710]
[gfsnode5][WARNIN] 3: (gsignal()+0x35) [0x34ae832925]
[gfsnode5][WARNIN] 4: (abort()+0x175) [0x34ae834105]
[gfsnode5][WARNIN] 5: /usr/lib64/libleveldb.so.1() [0x33bf034d71]
[gfsnode5][WARNIN] 6: (leveldb::DBImpl::Get(leveldb::ReadOptions const,
leveldb::Slice const, leveldb::Value*)+0x50) [0x33bf01a120]
[gfsnode5][WARNIN] 7: (LevelDBStore::_get_iterator()+0x41) [0x7773b1]
[gfsnode5][WARNIN] 8: (MonitorDBStore::exists(std::string const, std::string
const)+0x28) [0x5276f8]
[gfsnode5][WARNIN] 9: (Monitor::check_fsid()+0x73) [0x550bb3]
[gfsnode5][WARNIN] 10: (Monitor::mkfs(ceph::buffer::list)+0x47) [0x55f027]
[gfsnode5][WARNIN] 11: (main()+0x11d7) [0x521f97]
[gfsnode5][WARNIN] 12: (__libc_start_main()+0xfd) [0x34ae81ed1d]
[gfsnode5][WARNIN] 13: ceph-mon() [0x51fa79]
[gfsnode5][WARNIN] 2014-05-02 11:19:46.335854 7f22e8d737a0 -1 *** Caught signal
(Aborted) **
[gfsnode5][WARNIN] in thread 7f22e8d737a0
[gfsnode5][WARNIN]
[gfsnode5][WARNIN] ceph version 0.72.2
(a913ded2ff138aefb8cb84d347d72164099cfd60)
[gfsnode5][WARNIN] 1: ceph-mon() [0x808571]
[gfsnode5][WARNIN] 2: /lib64/libpthread.so.0() [0x34aec0f710]
[gfsnode5][WARNIN] 3: (gsignal()+0x35) [0x34ae832925]
[gfsnode5][WARNIN] 4: (abort()+0x175) [0x34ae834105]
[gfsnode5][WARNIN] 5: /usr/lib64/libleveldb.so.1() [0x33bf034d71]
[gfsnode5][WARNIN] 6: (leveldb::DBImpl::Get(leveldb::ReadOptions const,
leveldb::Slice const, leveldb::Value*)+0x50) [0x33bf01a120]
[gfsnode5][WARNIN] 7: (LevelDBStore::_get_iterator()+0x41) [0x7773b1]
[gfsnode5][WARNIN] 8: (MonitorDBStore::exists(std::string const, std::string
const)+0x28) [0x5276f8]
[gfsnode5][WARNIN] 9: (Monitor::check_fsid()+0x73) [0x550bb3]
[gfsnode5][WARNIN] 10: (Monitor::mkfs(ceph::buffer::list)+0x47) [0x55f027]
[gfsnode5][WARNIN] 11: (main()+0x11d7) [0x521f97]
[gfsnode5][WARNIN] 12: (__libc_start_main()+0xfd) [0x34ae81ed1d]
[gfsnode5][WARNIN] 13: ceph-mon() [0x51fa79]
[gfsnode5][WARNIN] NOTE: a copy of the executable, or `objdump -rdS
executable` is needed to interpret this.
