[ceph-users] ceph.conf not found
Hey, I get this error: gentooserver ~ # ceph -s 2018-01-04 14:38:35.390154 7f0a6bae8700 -1 Errors while parsing config file! 2018-01-04 14:38:35.390157 7f0a6bae8700 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory 2018-01-04 14:38:35.390158 7f0a6bae8700 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory 2018-01-04 14:38:35.390158 7f0a6bae8700 -1 parse_file: cannot open ceph.conf: (2) No such file or directory Error initializing cluster client: ObjectNotFound('error calling conf_read_file',) I don't have a ceph.conf because my cluster name is "home" and the guide at http://docs.ceph.com/docs/master/install/manual-deployment/ says to name the configuration file after the cluster name. What should I do to resolve this? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] permission denied, unable to bind socket
So I'm following the guide at http://docs.ceph.com/docs/master/install/manual-deployment/ ceph-mon@gentooserver.service fails. Jan 06 19:12:40 gentooserver systemd[1]: Started Ceph cluster monitor daemon. Jan 06 19:12:41 gentooserver ceph-mon[2674]: warning: unable to create /var/run/ceph: (13) Permission denied Jan 06 19:12:41 gentooserver ceph-mon[2674]: 2018-01-06 19:12:41.000507 7f3163008f80 -1 asok(0x563d0f97d2c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-mon.gentooserver.asok': (2) No such file or directory Jan 06 19:12:41 gentooserver ceph-mon[2674]: 2018-01-06 19:12:41.152781 7f3163008f80 -1 Processor -- bind unable to bind to [2001:1c:d64b:91c5:3a84:dfce:8546:998]:6789/0: (99) Cannot assign requested address Jan 06 19:12:41 gentooserver ceph-mon[2674]: 2018-01-06 19:12:41.152789 7f3163008f80 -1 Processor -- bind was unable to bind. Trying again in 5 seconds Jan 06 19:12:46 gentooserver ceph-mon[2674]: 2018-01-06 19:12:46.152982 7f3163008f80 -1 Processor -- bind unable to bind to [2001:1c:d64b:91c5:3a84:dfce:8546:998]:6789/0: (99) Cannot assign requested address Jan 06 19:12:46 gentooserver ceph-mon[2674]: 2018-01-06 19:12:46.152996 7f3163008f80 -1 Processor -- bind was unable to bind. Trying again in 5 seconds Jan 06 19:12:51 gentooserver ceph-mon[2674]: 2018-01-06 19:12:51.153218 7f3163008f80 -1 Processor -- bind unable to bind to [2001:1c:d64b:91c5:3a84:dfce:8546:998]:6789/0: (99) Cannot assign requested address Jan 06 19:12:51 gentooserver ceph-mon[2674]: 2018-01-06 19:12:51.153244 7f3163008f80 -1 Processor -- bind was unable to bind after 3 attempts: (99) Cannot assign requested address Jan 06 19:12:51 gentooserver ceph-mon[2674]: 2018-01-06 19:12:51.153250 7f3163008f80 -1 unable to bind monitor to [2001:1c:d64b:91c5:3a84:dfce:8546:998]:6789/0 Jan 06 19:12:51 gentooserver systemd[1]: ceph-mon@gentooserver.service: Main process exited, code=exited, status=1/FAILURE Jan 06 19:12:51 gentooserver systemd[1]: ceph-mon@gentooserver.service: Unit entered failed state. Jan 06 19:12:51 gentooserver systemd[1]: ceph-mon@gentooserver.service: Failed with result 'exit-code'. Jan 06 19:13:01 gentooserver systemd[1]: ceph-mon@gentooserver.service: Service hold-off time over, scheduling restart. Jan 06 19:13:01 gentooserver systemd[1]: Stopped Ceph cluster monitor daemon. Jan 06 19:13:01 gentooserver systemd[1]: ceph-mon@gentooserver.service: Start request repeated too quickly. Jan 06 19:13:01 gentooserver systemd[1]: Failed to start Ceph cluster monitor daemon. Jan 06 19:13:01 gentooserver systemd[1]: ceph-mon@gentooserver.service: Unit entered failed state. Jan 06 19:13:01 gentooserver systemd[1]: ceph-mon@gentooserver.service: Failed with result 'exit-code'. cat /etc/ceph/ceph.conf [global] fsid = a736559a-92d1-483e-9289-d2c7feed510f ms bind ipv6 = true mon initial members = gentooserver [mon.mona] host = gentooserver mon addr = [2001:1c:d64b:91c5:3a84:dfce:8546:9982] [osd] osd journal size = 1 I'm not sure if the problem is the permissions error, or the IP address appearing to get truncated in the output, or both. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] permission denied, unable to bind socket
Ok I fixed the address error. The service is able to start now. ceph -s hangs though. gentooserver ~ # ceph -s ^CError EINTR: problem getting command descriptions from mon. I'm not sure how to fix the permissions issue. /var/run/ceph is a temporary directory so I can't just chown it. On Sat, Jan 6, 2018 at 7:40 PM, Nathan Dehnel wrote: > So I'm following the guide at http://docs.ceph.com/docs/ > master/install/manual-deployment/ > > ceph-mon@gentooserver.service fails. > > Jan 06 19:12:40 gentooserver systemd[1]: Started Ceph cluster monitor > daemon. > Jan 06 19:12:41 gentooserver ceph-mon[2674]: warning: unable to create > /var/run/ceph: (13) Permission denied > Jan 06 19:12:41 gentooserver ceph-mon[2674]: 2018-01-06 19:12:41.000507 > 7f3163008f80 -1 asok(0x563d0f97d2c0) AdminSocketConfigObs::init: failed: > AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to > '/var/run/ceph/ceph-mon.gentooserver.asok': (2) No such file or directory > Jan 06 19:12:41 gentooserver ceph-mon[2674]: 2018-01-06 19:12:41.152781 > 7f3163008f80 -1 Processor -- bind unable to bind to > [2001:1c:d64b:91c5:3a84:dfce:8546:998]:6789/0: (99) Cannot assign > requested address > Jan 06 19:12:41 gentooserver ceph-mon[2674]: 2018-01-06 19:12:41.152789 > 7f3163008f80 -1 Processor -- bind was unable to bind. Trying again in 5 > seconds > Jan 06 19:12:46 gentooserver ceph-mon[2674]: 2018-01-06 19:12:46.152982 > 7f3163008f80 -1 Processor -- bind unable to bind to > [2001:1c:d64b:91c5:3a84:dfce:8546:998]:6789/0: (99) Cannot assign > requested address > Jan 06 19:12:46 gentooserver ceph-mon[2674]: 2018-01-06 19:12:46.152996 > 7f3163008f80 -1 Processor -- bind was unable to bind. Trying again in 5 > seconds > Jan 06 19:12:51 gentooserver ceph-mon[2674]: 2018-01-06 19:12:51.153218 > 7f3163008f80 -1 Processor -- bind unable to bind to > [2001:1c:d64b:91c5:3a84:dfce:8546:998]:6789/0: (99) Cannot assign > requested address > Jan 06 19:12:51 gentooserver ceph-mon[2674]: 2018-01-06 19:12:51.153244 > 7f3163008f80 -1 Processor -- bind was unable to bind after 3 attempts: > (99) Cannot assign requested address > Jan 06 19:12:51 gentooserver ceph-mon[2674]: 2018-01-06 19:12:51.153250 > 7f3163008f80 -1 unable to bind monitor to [2001:1c:d64b:91c5:3a84:dfce: > 8546:998]:6789/0 > Jan 06 19:12:51 gentooserver systemd[1]: ceph-mon@gentooserver.service: > Main process exited, code=exited, status=1/FAILURE > Jan 06 19:12:51 gentooserver systemd[1]: ceph-mon@gentooserver.service: > Unit entered failed state. > Jan 06 19:12:51 gentooserver systemd[1]: ceph-mon@gentooserver.service: > Failed with result 'exit-code'. > Jan 06 19:13:01 gentooserver systemd[1]: ceph-mon@gentooserver.service: > Service hold-off time over, scheduling restart. > Jan 06 19:13:01 gentooserver systemd[1]: Stopped Ceph cluster monitor > daemon. > Jan 06 19:13:01 gentooserver systemd[1]: ceph-mon@gentooserver.service: > Start request repeated too quickly. > Jan 06 19:13:01 gentooserver systemd[1]: Failed to start Ceph cluster > monitor daemon. > Jan 06 19:13:01 gentooserver systemd[1]: ceph-mon@gentooserver.service: > Unit entered failed state. > Jan 06 19:13:01 gentooserver systemd[1]: ceph-mon@gentooserver.service: > Failed with result 'exit-code'. > > cat /etc/ceph/ceph.conf > [global] > fsid = a736559a-92d1-483e-9289-d2c7feed510f > ms bind ipv6 = true > mon initial members = gentooserver > > [mon.mona] > host = gentooserver > mon addr = [2001:1c:d64b:91c5:3a84:dfce:8546:9982] > > [osd] > osd journal size = 1 > > I'm not sure if the problem is the permissions error, or the IP address > appearing to get truncated in the output, or both. > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] ceph command hangs
I'm doing a manual setup following http://docs.ceph.com/docs/master/install/manual-deployment/ The ceph command hangs until I kill it. I have 1 monitor service started. == gentooserver ~ # ceph -s ^CError EINTR: problem getting command descriptions from mon. = gentooserver ~ # emerge -pqv ceph [ebuild R ] sys-cluster/ceph-12.2.1 USE="mgr radosgw ssl systemd tcmalloc -babeltrace -cephfs -fuse -jemalloc -ldap -lttng -nss -static-libs {-test} -xfs -zfs" CPU_FLAGS_X86="sse sse2 sse3 sse4_1 sse4_2 ssse3" PYTHON_TARGETS="python2_7 python3_5 -python3_4 -python3_6" == gentooserver ~ # cat /etc/ceph/ceph.conf [global] cluster = ceph fsid = a736559a-92d1-483e-9289-d2c7feed510f ms bind ipv6 = true #public network = 2001:1c:d64b:91c5:3a84:dfce:8546:9982/0 auth cluster required = none auth service required = none auth client required = none [mon] mon initial members = mon0 mon host = gentooserver mon addr = [2001:1c:d64b:91c5:3a84:dfce:8546:9982]:6789 [mon.mon0] host = gentooserver mon addr = [2001:1c:d64b:91c5:3a84:dfce:8546:9982]:6789 [osd] osd journal size = 1 osd crush chooseleaf type = 0 == gentooserver ~ # monmaptool --print /tmp/monmap monmaptool: monmap file /tmp/monmap epoch 0 fsid a736559a-92d1-483e-9289-d2c7feed510f last_changed 2018-01-14 16:50:59.838277 created 2018-01-14 16:50:59.838277 0: [2001:1c:d64b:91c5:3a84:dfce:8546:9982]:6789/0 mon.mon0 == gentooserver ~ # systemctl status ceph-mon@mon0 | cat ● ceph-mon@mon0.service - Ceph cluster monitor daemon Loaded: loaded (/usr/lib/systemd/system/ceph-mon@.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2018-01-16 14:50:18 CST; 17min ago Main PID: 75938 (ceph-mon) CGroup: /system.slice/system-ceph\x2dmon.slice/ceph-mon@mon0.service └─75938 /usr/bin/ceph-mon -f --cluster ceph --id mon0 --setuser ceph --setgroup ceph Jan 16 14:50:18 gentooserver systemd[1]: Started Ceph cluster monitor daemon. Jan 16 14:50:18 gentooserver ceph-mon[75938]: 2018-01-16 14:50:18.977494 7ff07d4cef80 -1 distro_detect - can't detect distro_version === gentooserver ~ # cat /var/log/ceph/ceph.log 2018-01-16 14:50:18.977541 mon.mon0 mon.0 [2001:1c:d64b:91c5:3a84:dfce:8546:9982]:6789/0 1 : cluster [INF] mon.mon0@0 won leader election with quorum 0 2018-01-16 14:50:18.977656 mon.mon0 mon.0 [2001:1c:d64b:91c5:3a84:dfce:8546:9982]:6789/0 2 : cluster [INF] monmap e1: 1 mons at {mon0=[2001:1c:d64b:91c5:3a84:dfce:8546:9982]:6789/0} --- gentooserver ~ # cat /var/log/ceph/ceph-mon.mon0.log 2018-01-16 14:50:18.760533 7ff07d4cef80 0 set uid:gid to 110:239 (ceph:ceph) 2018-01-16 14:50:18.760549 7ff07d4cef80 0 ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable), process (unknown), pid 75938 2018-01-16 14:50:18.760591 7ff07d4cef80 0 pidfile_write: ignore empty --pid-file 2018-01-16 14:50:18.765642 7ff07d4cef80 0 load: jerasure load: lrc load: isa 2018-01-16 14:50:18.765702 7ff07d4cef80 0 set rocksdb option compression = kNoCompression 2018-01-16 14:50:18.765709 7ff07d4cef80 0 set rocksdb option write_buffer_size = 33554432 2018-01-16 14:50:18.765722 7ff07d4cef80 0 set rocksdb option compression = kNoCompression 2018-01-16 14:50:18.765726 7ff07d4cef80 0 set rocksdb option write_buffer_size = 33554432 2018-01-16 14:50:18.765798 7ff07d4cef80 4 rocksdb: RocksDB version: 5.4.0 2018-01-16 14:50:18.765804 7ff07d4cef80 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2018-01-16 14:50:18.765806 7ff07d4cef80 4 rocksdb: Compile date Dec 14 2017 2018-01-16 14:50:18.765808 7ff07d4cef80 4 rocksdb: DB SUMMARY 2018-01-16 14:50:18.765837 7ff07d4cef80 4 rocksdb: CURRENT file: CURRENT 2018-01-16 14:50:18.765840 7ff07d4cef80 4 rocksdb: IDENTITY file: IDENTITY 2018-01-16 14:50:18.765843 7ff07d4cef80 4 rocksdb: MANIFEST file: MANIFEST-11 size: 210 Bytes 2018-01-16 14:50:18.765845 7ff07d4cef80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-mon0/store.db dir, Total Num: 3, files: 04.sst 07.sst 10.sst 2018-01-16 14:50:18.765846 7ff07d4cef80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-mon0/store.db: 12.log size: 9161 ; 2018-01-16 14:50:18.765847 7ff07d4cef80 4 rocksdb: Options.error_if_exists: 0 2018-01-16 14:50:18.765848 7ff07d4cef80 4 rocksdb: Options.create_if_missing: 0 2018-01-16 14:50:18.765848 7ff07d4cef80 4 rocksdb: Options.paranoid_checks: 1 2018-01-16 14:50:18.765849 7ff07d4cef80 4 rocksdb: Options.env: 0x55bc0ff83a60 2018-01-16 14:50:18.765849 7ff07d4cef80 4 rocksdb: Options.info_log: 0x55bc10a53ae0 2018-01-16 14:50:18.765850 7ff07d4cef80 4 rocksdb: Options.max_op
Re: [ceph-users] ceph command hangs
11:11.984458 7f9579ffb700 1 -- [2001:1c:d64b:91c5:3a84:dfce:8546:9982]:0/285566692 --> [2001:1c:d64b:91c5:3a84:dfce:8546:9982]:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f955c0041d0 con 0 2018-01-18 21:11:11.984678 7f9579ffb700 1 -- [2001:1c:d64b:91c5:3a84:dfce:8546:9982]:0/285566692 <== mon.0 [2001:1c:d64b:91c5:3a84:dfce:8546:9982]:6789/0 2 mgrmap(e 1) v1 103+0+0 (706778617 0 0) 0x7f956c000fd0 con 0x7f9560007290 ^CError EINTR: problem getting command descriptions from mon. 2018-01-18 21:11:41.999310 7f9571ffb700 1 -- [2001:1c:d64b:91c5:3a84:dfce:8546:9982]:0/285566692 >> [2001:1c:d64b:91c5:3a84:dfce:8546:9982]:6789/0 conn(0x7f9560007290 :-1 s=STATE_OPEN pgs=5 cs=1 l=1).mark_down 2018-01-18 21:11:41.999382 7f9571ffb700 1 -- [2001:1c:d64b:91c5:3a84:dfce:8546:9982]:0/285566692 shutdown_connections 2018-01-18 21:11:41.999464 7f9571ffb700 1 -- [2001:1c:d64b:91c5:3a84:dfce:8546:9982]:0/285566692 shutdown_connections 2018-01-18 21:11:41.999488 7f9571ffb700 1 -- [2001:1c:d64b:91c5:3a84:dfce:8546:9982]:0/285566692 wait complete. 2018-01-18 21:11:41.999494 7f9571ffb700 1 -- [2001:1c:d64b:91c5:3a84:dfce:8546:9982]:0/285566692 >> [2001:1c:d64b:91c5:3a84:dfce:8546:9982]:0/285566692 conn(0x7f957c118c90 :-1 s=STATE_NONE pgs=0 cs=0 l=0).mark_down To my knowledge there are no firewall settings that would cause problems. I'm running ceph -s on the monitor node. On Tue, Jan 16, 2018 at 8:05 PM, Jean-Charles Lopez wrote: > Hi Nathan, > > I would have place the mon_host parameter and assigned it the IP address > for your monitor host in the global section so that the client (ceph -s > command). > > Have you also checked your firewall setup on your MON box? > > To help diagnose you can also use ceph -s --debug-ms=1 so you can follow > the network exchange between your client machine and the MON. > > Regards > JC > > > On Jan 16, 2018, at 13:24, Nathan Dehnel wrote: > > I'm doing a manual setup following http://docs.ceph. > com/docs/master/install/manual-deployment/ > > The ceph command hangs until I kill it. I have 1 monitor service started. > == > gentooserver ~ # ceph -s > ^CError EINTR: problem getting command descriptions from mon. > = > gentooserver ~ # emerge -pqv ceph > [ebuild R ] sys-cluster/ceph-12.2.1 USE="mgr radosgw ssl systemd > tcmalloc -babeltrace -cephfs -fuse -jemalloc -ldap -lttng -nss -static-libs > {-test} -xfs -zfs" CPU_FLAGS_X86="sse sse2 sse3 sse4_1 sse4_2 ssse3" > PYTHON_TARGETS="python2_7 python3_5 -python3_4 -python3_6" > == > gentooserver ~ # cat /etc/ceph/ceph.conf > [global] > cluster = ceph > fsid = a736559a-92d1-483e-9289-d2c7feed510f > ms bind ipv6 = true > #public network = 2001:1c:d64b:91c5:3a84:dfce:8546:9982/0 > auth cluster required = none > auth service required = none > auth client required = none > > [mon] > mon initial members = mon0 > mon host = gentooserver > mon addr = [2001:1c:d64b:91c5:3a84:dfce:8546:9982]:6789 > > > [mon.mon0] > host = gentooserver > mon addr = [2001:1c:d64b:91c5:3a84:dfce:8546:9982]:6789 > > [osd] > osd journal size = 1 > osd crush chooseleaf type = 0 > == > gentooserver ~ # monmaptool --print /tmp/monmap > monmaptool: monmap file /tmp/monmap > epoch 0 > fsid a736559a-92d1-483e-9289-d2c7feed510f > last_changed 2018-01-14 16:50:59.838277 > created 2018-01-14 16:50:59.838277 > 0: [2001:1c:d64b:91c5:3a84:dfce:8546:9982]:6789/0 mon.mon0 > == > gentooserver ~ # systemctl status ceph-mon@mon0 | cat > ● ceph-mon@mon0.service - Ceph cluster monitor daemon >Loaded: loaded (/usr/lib/systemd/system/ceph-mon@.service; enabled; > vendor preset: disabled) >Active: active (running) since Tue 2018-01-16 14:50:18 CST; 17min ago > Main PID: 75938 (ceph-mon) >CGroup: /system.slice/system-ceph\x2dmon.slice/ceph-mon@mon0.service >└─75938 /usr/bin/ceph-mon -f --cluster ceph --id mon0 --setuser > ceph --setgroup ceph > > Jan 16 14:50:18 gentooserver systemd[1]: Started Ceph cluster monitor > daemon. > Jan 16 14:50:18 gentooserver ceph-mon[75938]: 2018-01-16 14:50:18.977494 > 7ff07d4cef80 -1 distro_detect - can't detect distro_version > === > gentooserver ~ # cat /var/log/ceph/ceph.log > 2018-01-16 14:50:18.977541 mon.mon0 mon.0 > [2001:1c:d64b:91c5:3a84:dfce:8546:9982]:6789/0 > 1 : cluster [INF] mon.mon0@0 won leader election with quorum 0 > 2018-01-16 14:50:18.977656 mon.mon0 mon.0 > [2
[ceph-users] ceph-volume raw disks
The doc at http://docs.ceph.com/docs/master/ceph-volume/lvm/prepare/#ceph-volume-lvm-prepare says I can pass a physical device to ceph-volume. But when I try to do that: gentooserver ~ # ceph-volume lvm create --bluestore --data /dev/sdb usage: ceph-volume lvm create [-h] [--journal JOURNAL] --data DATA [--journal-size GB] [--bluestore] [--filestore] [--osd-id OSD_ID] [--osd-fsid OSD_FSID] ceph-volume lvm create: error: Logical volume must be specified as 'volume_group/logical_volume' but got: /dev/sdb Am I missing something? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Fwd: ceph-volume raw disks
-- Forwarded message -- From: Nathan Dehnel Date: Thu, Jan 25, 2018 at 9:49 PM Subject: Re: [ceph-users] ceph-volume raw disks To: David Turner >Did you wipe all of the existing partitions and such first? I tried it both before and after creating an lvm physical partition. >Which version of ceph? 12.2.1 gentooserver ~ # ceph-volume lvm zap /dev/sdb usage: ceph-volume lvm [-h] ceph-volume lvm: error: unrecognized arguments: zap /dev/sdb ??? On Thu, Jan 25, 2018 at 9:42 PM, David Turner wrote: > Did you wipe all of the existing partitions and such first? Which version > of ceph? The below commands are what I ran to re-add my osds as bluestore > after moving all data off of them. > > ceph-volume lvm zap /dev/sdb > ceph-volume lvm create --bluestore --data /dev/sdb > > On Thu, Jan 25, 2018 at 9:41 PM Nathan Dehnel wrote: > >> The doc at http://docs.ceph.com/docs/master/ceph-volume/lvm/prepare/ >> #ceph-volume-lvm-prepare says I can pass a physical device to >> ceph-volume. But when I try to do that: >> >> gentooserver ~ # ceph-volume lvm create --bluestore --data /dev/sdb >> usage: ceph-volume lvm create [-h] [--journal JOURNAL] --data DATA >> [--journal-size GB] [--bluestore] >> [--filestore] >> [--osd-id OSD_ID] [--osd-fsid OSD_FSID] >> ceph-volume lvm create: error: Logical volume must be specified as >> 'volume_group/logical_volume' but got: /dev/sdb >> >> Am I missing something? >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] (no subject)
Trying to create an OSD: gentooserver ~ # ceph-volume lvm create --data /dev/sdb Running command: ceph-authtool --gen-print-key Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e70500fe-0d51-48c3-a607-414957886726 Running command: vgcreate --force --yes ceph-a736559a-92d1-483e-9289-d2c7feed510f /dev/sdb stdout: Volume group "ceph-a736559a-92d1-483e-9289-d2c7feed510f" successfully created Running command: lvcreate --yes -l 100%FREE -n osd-block-e70500fe-0d51-48c3-a607-414957886726 ceph-a736559a-92d1-483e-9289-d2c7feed510f stdout: Logical volume "osd-block-e70500fe-0d51-48c3-a607-414957886726" created. Running command: ceph-authtool --gen-print-key Running command: mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0 Running command: chown -R ceph:ceph /dev/dm-0 Running command: ln -s /dev/ceph-a736559a-92d1-483e-9289-d2c7feed510f/osd-block-e70500fe-0d51-48c3-a607-414957886726 /var/lib/ceph/osd/ceph-0/block Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap stderr: got monmap epoch 1 Running command: ceph-authtool /var/lib/ceph/osd/ceph-0/keyring --create-keyring --name osd.0 --add-key AQBEZqRalRoRCBAA03R6VshykLcZjMgQnFKDtg== stdout: creating /var/lib/ceph/osd/ceph-0/keyring stdout: added entity osd.0 auth auth(auid = 18446744073709551615 key=AQBEZqRalRoRCBAA03R6VshykLcZjMgQnFKDtg== with 0 caps) Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/ Running command: ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid e70500fe-0d51-48c3-a607-414957886726 --setuser ceph --setgroup ceph --> ceph-volume lvm prepare successful for: /dev/sdb Running command: ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-a736559a-92d1-483e-9289-d2c7feed510f/osd-block-e70500fe-0d51-48c3-a607-414957886726 --path /var/lib/ceph/osd/ceph-0 Running command: ln -snf /dev/ceph-a736559a-92d1-483e-9289-d2c7feed510f/osd-block-e70500fe-0d51-48c3-a607-414957886726 /var/lib/ceph/osd/ceph-0/block Running command: chown -R ceph:ceph /dev/dm-0 Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 Running command: systemctl enable ceph-volume@lvm-0-e70500fe-0d51-48c3-a607-414957886726 stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-e70500fe-0d51-48c3-a607-414957886726.service → /lib/systemd/system/ceph-volume@.service. Running command: systemctl start ceph-osd@0 stderr: Job for ceph-osd@0.service failed because the control process exited with error code. See "systemctl status ceph-osd@0.service" and "journalctl -xe" for details. --> Was unable to complete a new OSD, will rollback changes --> OSD will be fully purged from the cluster, because the ID was generated Running command: ceph osd purge osd.0 --yes-i-really-mean-it stderr: purged osd.0 --> RuntimeError: command returned non-zero exit status: 1 journalctl -xe -- Unit ceph-osd@0.service has begun starting up. Mar 10 17:14:34 gentooserver systemd[3977]: ceph-osd@0.service: Failed to execute command: No such file or directory Mar 10 17:14:34 gentooserver systemd[3977]: ceph-osd@0.service: Failed at step EXEC spawning /usr/lib/ceph/ceph-osd-prestart.sh: No such file or directory -- Subject: Process /usr/lib/ceph/ceph-osd-prestart.sh could not be executed -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- The process /usr/lib/ceph/ceph-osd-prestart.sh could not be executed and failed. -- -- The error number returned by this process is 2. Mar 10 17:14:34 gentooserver systemd[1]: ceph-osd@0.service: Control process exited, code=exited status=203 Mar 10 17:14:34 gentooserver systemd[1]: ceph-osd@0.service: Failed with result 'exit-code'. Mar 10 17:14:34 gentooserver systemd[1]: Failed to start Ceph object storage daemon osd.0. -- Subject: Unit ceph-osd@0.service has failed -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit ceph-osd@0.service has failed. -- -- The result is RESULT. Why is this file missing? Should I file a bug with my distro's packager? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] PG numbers don't add up?
I try to add a data pool: OSD_STAT USED AVAIL TOTAL HB_PEERSPG_SUM PRIMARY_PG_SUM 9 1076M 930G 931G [0,1,2,3,4,5,6,7,8]128 5 8 1076M 930G 931G [0,1,2,3,4,5,6,7,9]128 14 7 1076M 930G 931G [0,1,2,3,4,5,6,8,9]128 14 6 1076M 930G 931G [0,1,2,3,4,5,7,8,9]128 19 5 1076M 930G 931G [0,1,2,3,4,6,7,8,9]128 15 4 1076M 930G 931G [0,1,2,3,5,6,7,8,9]128 17 0 1076M 930G 931G [1,2,3,4,5,6,7,8,9]128 16 1 1076M 930G 931G [0,2,3,4,5,6,7,8,9]128 8 2 1076M 930G 931G [0,1,3,4,5,6,7,8,9]128 8 3 1076M 930G 931G [0,1,2,4,5,6,7,8,9]128 12 sum 10765M 9304G 9315G I try to add a metadata pool: sum 0 0 0 0 0 0 0 0 OSD_STAT USED AVAIL TOTAL HB_PEERSPG_SUM PRIMARY_PG_SUM 9 1076M 930G 931G [0,1,2,3,4,5,6,7,8] 73 73 8 1076M 930G 931G [0,1,2,3,4,5,6,7,9] 40 40 7 1076M 930G 931G [0,1,2,3,4,5,6,8,9] 56 56 6 1076M 930G 931G [0,1,2,3,4,5,7,8,9] 42 42 5 1076M 930G 931G [0,1,2,3,4,6,7,8,9] 54 54 4 1076M 930G 931G [0,1,2,3,5,6,7,8,9] 59 59 0 1076M 930G 931G [1,2,3,4,5,6,7,8,9] 38 38 1 1076M 930G 931G [0,2,3,4,5,6,7,8,9] 57 57 2 1076M 930G 931G [0,1,3,4,5,6,7,8,9] 45 45 3 1076M 930G 931G [0,1,2,4,5,6,7,8,9] 48 48 sum 10766M 9304G 9315G I try to add both pools: Error ERANGE: pg_num 128 size 10 would mean 2816 total pgs, which exceeds max 2000 (mon_max_pg_per_osd 200 * num_in_osds 10) That's over a thousand more PGs than both pools combined. Where are they coming from? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] "unable to connect to cluster" after monitor IP change
gentooserver ~ # ceph-mon -i mon0 --extract-monmap /tmp/monmap 2018-04-06 15:38:10.863444 7f8aa2b72f80 -1 wrote monmap to /tmp/monmap gentooserver ~ # monmaptool --print /tmp/monmap monmaptool: monmap file /tmp/monmap epoch 3 fsid a736559a-92d1-483e-9289-d2c7feed510f last_changed 2018-04-06 14:53:12.892574 created 2018-04-06 14:52:18.190509 0: [2605:6000:1020:2056:7d79:3f08:ee64:2aa3]:6789/0 mon.mon0 This is the monmap I injected into my monitor. gentooserver ~ # systemctl status ceph-mon@mon0 ● ceph-mon@mon0.service - Ceph cluster monitor daemon Loaded: loaded (/lib/systemd/system/ceph-mon@.service; indirect; vendor preset: disabled) Active: active (running) since Fri 2018-04-06 15:47:51 CDT; 16s ago Main PID: 4362 (ceph-mon) CGroup: /system.slice/system-ceph\x2dmon.slice/ceph-mon@mon0.service └─4362 /usr/bin/ceph-mon -f --cluster ceph --id mon0 --setuser ceph --setgroup ceph Apr 06 15:47:51 gentooserver systemd[1]: Started Ceph cluster monitor daemon. Apr 06 15:47:51 gentooserver ceph-mon[4362]: 2018-04-06 15:47:51.841218 7f5824d44f80 -1 distro_detect - can't detect distro_version gentooserver ~ # systemctl status ceph-mgr@mgr0 ● ceph-mgr@mgr0.service - Ceph cluster manager daemon Loaded: loaded (/lib/systemd/system/ceph-mgr@.service; indirect; vendor preset: disabled) Active: active (running) since Fri 2018-04-06 15:10:01 CDT; 38min ago Main PID: 3807 (ceph-mgr) CGroup: /system.slice/system-ceph\x2dmgr.slice/ceph-mgr@mgr0.service └─3807 /usr/bin/ceph-mgr -f --cluster ceph --id mgr0 --setuser ceph --setgroup ceph Apr 06 15:10:01 gentooserver systemd[1]: Started Ceph cluster manager daemon. gentooserver ~ # systemctl status ceph-mds@mds0 ● ceph-mds@mds0.service - Ceph metadata server daemon Loaded: loaded (/lib/systemd/system/ceph-mds@.service; disabled; vendor preset: disabled) Drop-In: /etc/systemd/system/ceph-mds@.service.d └─00gentoo.conf Active: active (running) since Fri 2018-04-06 15:10:25 CDT; 38min ago Main PID: 3827 (ceph-mds) CGroup: /system.slice/system-ceph\x2dmds.slice/ceph-mds@mds0.service └─3827 /usr/bin/ceph-mds -f --cluster ceph --id mds0 --setuser ceph --setgroup ceph Apr 06 15:10:25 gentooserver systemd[1]: Started Ceph metadata server daemon. Apr 06 15:10:25 gentooserver ceph-mds[3827]: starting mds.mds0 at - All the daemons report they are running. gentooserver ~ # ceph daemon mon.mon0 mon_status { "name": "mon0", "rank": 0, "state": "leader", "election_epoch": 39, "quorum": [ 0 ], "features": { "required_con": "153140804152475648", "required_mon": [ "kraken", "luminous" ], "quorum_con": "2305244844532236283", "quorum_mon": [ "kraken", "luminous" ] }, "outside_quorum": [], "extra_probe_peers": [], "sync_provider": [], "monmap": { "epoch": 3, "fsid": "a736559a-92d1-483e-9289-d2c7feed510f", "modified": "2018-04-06 14:53:12.892574", "created": "2018-04-06 14:52:18.190509", "features": { "persistent": [ "kraken", "luminous" ], "optional": [] }, "mons": [ { "rank": 0, "name": "mon0", "addr": "[2605:6000:1020:2056:7d79:3f08:ee64:2aa3]:6789/0", "public_addr": "[2605:6000:1020:2056:7d79:3f08:ee64:2aa3]:6789/0" } ] }, "feature_map": { "mon": { "group": { "features": "0x1ffddff8eea4fffb", "release": "luminous", "num": 1 } } } } gentooserver ~ # cat /etc/ceph/ceph.conf [global] cluster = ceph fsid = a736559a-92d1-483e-9289-d2c7feed510f ms bind ipv6 = true auth cluster required = none auth service required = none auth client required = none mon host = gentooserver mon addr = [2605:6000:1020:2056:7d79:3f08:ee64:2aa3]:6789 mon pg warn max per osd = 300 [mon] mon initial members = mon0 mon host = gentooserver mon addr = [2605:6000:1020:2056:7d79:3f08:ee64:2aa3]:6789 mon pg warn max per osd = 300 mon allow pool delete = true [mon.mon0] host = gentooserver mon addr = [2605:6000:1020:2056:7d79:3f08:ee64:2aa3]:6789 [osd] osd journal size = 1 osd crush chooseleaf type = 0 host = gentooserver osd pool default size = 3 osd pool default min size = 2 [mds.mds0] host = gentooserver ceph -s times out: gentooserver ~ # ceph -s 2018-04-06 15:58:29.861647 7f5e7f891700 0 monclient(hunting): authenticate timed out after 300 2018-04-06 15:58:29.861672 7f5e7f891700 0 librados: client.admin authentication error (110) Connection timed out [errno 110] error connecting to the cluster The cluster was working before the IP address changed. HELP ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.cep