Re: [ceph-users] firefly osds stuck in state booting
Hi Karan ,
Thanks .. that did the trick ..
The magic word was "in"
regarding rep size . I have adjusted them
my settings are
--snip--
osd pool default size =
2
osd pool default min size =
1
osd pool default pg num =
100
osd pool default pgp num = 100
--snip--
# Also in the meantime I had chance to play with ceph-deploy script too.
# Maybe it was me or probably it is a bug . I have tried twice and
everytime I have hit this
As I said before I'm using a directory as this is a test installation .
ceph-deploy osd prepare ceph2:/ceph2:/ceph2/journald <=== Works
but
--snip--
ceph-deploy osd activate
ceph2:/ceph2:/ceph2/journald
[ceph_deploy.conf][DEBUG ] found configuration file at:
/home/ceph/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.9): /usr/bin/ceph-deploy osd
activate ceph2:/ceph2:/ceph2/journald
[ceph_deploy.osd][DEBUG ] Activating cluster ceph disks
ceph2:/ceph2:/ceph2/journald
[ceph2][DEBUG ] connected to host: ceph2
[ceph2][DEBUG ] detect platform information from remote host
[ceph2][DEBUG ] detect machine type
[ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final
[ceph_deploy.osd][DEBUG ] activating host ceph2 disk /ceph2
[ceph_deploy.osd][DEBUG ] will use init type: sysvinit
[ceph2][INFO ] Running command: sudo ceph-disk-activate --mark-init
sysvinit --mount /ceph2
[ceph2][WARNIN] got monmap epoch 2
[ceph2][WARNIN] 2014-07-28 11:47:04.733204 7f08d1c667a0 -1 journal
FileJournal::_open: disabling aio for non-block journal. Use
journal_force_aio to force
use of aio anyway
[ceph2][WARNIN] 2014-07-28 11:47:04.733400 7f08d1c667a0 -1 journal check:
ondisk fsid ---- doesn't match expected
4795daff-
d63f-415b-9824-75f0863eb14f, invalid (someone else's?) journal
[ceph2][WARNIN] 2014-07-28 11:47:04.796835 7f08d1c667a0 -1 journal
FileJournal::_open: disabling aio for non-block journal. Use
journal_force_aio to force
use of aio anyway
[ceph2][WARNIN] 2014-07-28 11:47:04.798944 7f08d1c667a0 -1
filestore(/ceph2) could not find 23c2fcde/osd_superblock/0//-1 in index:
(2) No such file or dir
ectory
[ceph2][WARNIN] 2014-07-28 11:47:04.874282 7f08d1c667a0 -1 created object
store /ceph2 journal /ceph2/journal for osd.1 fsid
109507ab-adf1-4eb6-aacf-092549
4e3882
[ceph2][WARNIN] 2014-07-28 11:47:04.874474 7f08d1c667a0 -1 auth: error
reading file: /ceph2/keyring: can't open /ceph2/keyring: (2) No such file
or directo
ry
[ceph2][WARNIN] 2014-07-28 11:47:04.875209 7f08d1c667a0 -1 created new key
in keyring /ceph2/keyring
[ceph2][WARNIN] added key for osd.1
[ceph2][WARNIN] ceph-disk: Error: unable to create symlink
/var/lib/ceph/osd/ceph-1 -> /ceph2
[ceph2][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command:
ceph-disk-activate --mark-init sysvinit --mount /ceph2
--snip--
It turns out ceph-deploy does not create the directory /var/lib/ceph/osd
and if I create them everything works.
Cheers
On Mon, Jul 28, 2014 at 9:09 AM, Karan Singh wrote:
> The output that you have provided says that OSDs are not IN , Try the below
>
> ceph osd in osd.0
> ceph osd in osd.1
>
> service ceph start osd.0
> service ceph start osd.1
>
> If you have 1 more host with 1 disk , add it , starting Ceph Firefly
> default rep size is 3
>
>
> - Karan -
>
> On 27 Jul 2014, at 11:17, 10 minus wrote:
>
> Hi Sage,
>
> I have dropped all unset .. and even restarted the osd
> No dice .. OSDs are still stuck .
>
>
>
> --snip--
> ceph daemon osd.0 status│rtt
> min/avg/max/mdev = 0.095/0.120/0.236/0.015
> ms
>
> { "cluster_fsid":
> "99babb8f-c880-4b32-a227-94aa483d4871",
> │[root@ceph2 ~]# ceph daemon osd.1
> status
>
> "osd_fsid":
> "1ad28bde-c23c-44ba-a3b7-0fd3372e",│{
> "cluster_fsid":
> "99babb8f-c880-4b32-a227-94aa483d4871",
>
> "whoami":
> 0, │
> "osd_fsid":
> "becc3252-6977-47d6-87af-7b1337e591d8",
>
> "state":
> "booting",
> │ "whoami":
> 1,
>
> "oldest_map":
> 1, │
> "state":
> "booting",
>
> "newest_map":
> 24, │
> "oldest_map":
> 1,
>
> "num_pgs":
> 0} │
> "newest_map":
> 21,
>
> --snip--
>
> --snip--
> ceph osd
> tree
>
> # idweight type name up/down reweight
> -1 2 root default
> -3 1 host ceph1
> 0 1 osd.0 down0
> -2 1 host ceph2
> 1 1 osd.1 down0
>
> --snip--
>
> --snip--
> ceph -s
> cluster 2929fa80-0841-4cb6-a133-90b2098fc802
> health HEALTH_WARN 192 pgs stuck inactive; 192 pgs stuck unclean
> monmap e2: 3 mons at {ceph0=
> 10.0.12.220:6789/0,ceph1=10.0.12.221:6789/0,ceph2=10.0.12.222:6789/0},
> election e
Re: [ceph-users] firefly osds stuck in state booting
The output that you have provided says that OSDs are not IN , Try the below
ceph osd in osd.0
ceph osd in osd.1
service ceph start osd.0
service ceph start osd.1
If you have 1 more host with 1 disk , add it , starting Ceph Firefly default
rep size is 3
- Karan -
On 27 Jul 2014, at 11:17, 10 minus wrote:
> Hi Sage,
>
> I have dropped all unset .. and even restarted the osd
> No dice .. OSDs are still stuck .
>
>
>
> --snip--
> ceph daemon osd.0 status│rtt
> min/avg/max/mdev = 0.095/0.120/0.236/0.015 ms
>
> { "cluster_fsid": "99babb8f-c880-4b32-a227-94aa483d4871",
>│[root@ceph2 ~]# ceph daemon osd.1 status
>
>
> "osd_fsid": "1ad28bde-c23c-44ba-a3b7-0fd3372e",
>│{ "cluster_fsid": "99babb8f-c880-4b32-a227-94aa483d4871",
>
>
> "whoami": 0,
>│ "osd_fsid": "becc3252-6977-47d6-87af-7b1337e591d8",
>
>
> "state": "booting",
>│ "whoami": 1,
>
>
> "oldest_map": 1,
>│ "state": "booting",
>
>
> "newest_map": 24,
>│ "oldest_map": 1,
>
>
> "num_pgs": 0}
>│ "newest_map": 21,
>
>
> --snip--
>
> --snip--
> ceph osd tree
>
> # idweight type name up/down reweight
> -1 2 root default
> -3 1 host ceph1
> 0 1 osd.0 down0
> -2 1 host ceph2
> 1 1 osd.1 down0
>
> --snip--
>
> --snip--
> ceph -s
> cluster 2929fa80-0841-4cb6-a133-90b2098fc802
> health HEALTH_WARN 192 pgs stuck inactive; 192 pgs stuck unclean
> monmap e2: 3 mons at
> {ceph0=10.0.12.220:6789/0,ceph1=10.0.12.221:6789/0,ceph2=10.0.12.222:6789/0},
> election epoch 50, quorum 0,1,2 ceph0,ceph1,ceph2
> osdmap e24: 2 osds: 0 up, 0 in
> pgmap v25: 192 pgs, 3 pools, 0 bytes data, 0 objects
> 0 kB used, 0 kB / 0 kB avail
> 192 creating
> --snip--
>
>
>
>
> On Sat, Jul 26, 2014 at 5:57 PM, Sage Weil wrote:
> On Sat, 26 Jul 2014, 10 minus wrote:
> > Hi,
> >
> > I just setup a test ceph installation on 3 node Centos 6.5 .
> > two of the nodes are used for hosting osds and the third acts as mon .
> >
> > Please note I'm using LVM so had to set up the osd using the manual install
> > guide.
> >
> > --snip--
> > ceph -s
> > cluster 2929fa80-0841-4cb6-a133-90b2098fc802
> > health HEALTH_WARN 192 pgs stuck inactive; 192 pgs stuck unclean;
> > noup,nodown,noout flag(s) set
> > monmap e2: 3 mons
> > at{ceph0=10.0.12.220:6789/0,ceph1=10.0.12.221:6789/0,ceph2=10.0.12.222:6789/0
> > }, election epoch 46, quorum 0,1,2 ceph0,ceph1,ceph2
> > osdmap e21: 2 osds: 0 up, 0 in
> > flags noup,nodown,noout
>
>
> Do 'ceph osd unset noup' and they should start up. You likely also want
> to clear nodown and noout as well.
>
> sage
>
>
> > pgmap v22: 192 pgs, 3 pools, 0 bytes data, 0 objects
> > 0 kB used, 0 kB / 0 kB avail
> > 192 creating
> > --snip--
> >
> > osd tree
> >
> > --snip--
> > ceph osd tree
> > # idweight type name up/down reweight
> > -1 2 root default
> > -3 1 host ceph1
> > 0 1 osd.0 down0
> > -2 1 host ceph2
> > 1 1 osd.1 down0
> > --snip--
> >
> > --snip--
> > ceph daemon osd.0 status
> > { "cluster_fsid": "99babb8f-c880-4b32-a227-94aa483d4871",
> > "osd_fsid": "1ad28bde-c23c-44ba-a3b7-0fd3372e",
> > "whoami": 0,
> > "state": "booting",
> >
Re: [ceph-users] firefly osds stuck in state booting
Hi Sage,
I have dropped all unset .. and even restarted the osd
No dice .. OSDs are still stuck .
--snip--
ceph daemon osd.0 status│rtt
min/avg/max/mdev = 0.095/0.120/0.236/0.015
ms
{ "cluster_fsid":
"99babb8f-c880-4b32-a227-94aa483d4871",
│[root@ceph2 ~]# ceph daemon osd.1
status
"osd_fsid":
"1ad28bde-c23c-44ba-a3b7-0fd3372e",│{
"cluster_fsid":
"99babb8f-c880-4b32-a227-94aa483d4871",
"whoami":
0, │
"osd_fsid":
"becc3252-6977-47d6-87af-7b1337e591d8",
"state":
"booting",
│ "whoami":
1,
"oldest_map":
1, │
"state":
"booting",
"newest_map":
24, │
"oldest_map":
1,
"num_pgs":
0} │
"newest_map":
21,
--snip--
--snip--
ceph osd
tree
# idweight type name up/down reweight
-1 2 root default
-3 1 host ceph1
0 1 osd.0 down0
-2 1 host ceph2
1 1 osd.1 down0
--snip--
--snip--
ceph -s
cluster 2929fa80-0841-4cb6-a133-90b2098fc802
health HEALTH_WARN 192 pgs stuck inactive; 192 pgs stuck unclean
monmap e2: 3 mons at {ceph0=
10.0.12.220:6789/0,ceph1=10.0.12.221:6789/0,ceph2=10.0.12.222:6789/0},
election epoch 50, quorum 0,1,2 ceph0,ceph1,ceph2
osdmap e24: 2 osds: 0 up, 0 in
pgmap v25: 192 pgs, 3 pools, 0 bytes data, 0 objects
0 kB used, 0 kB / 0 kB avail
192 creating
--snip--
On Sat, Jul 26, 2014 at 5:57 PM, Sage Weil wrote:
> On Sat, 26 Jul 2014, 10 minus wrote:
> > Hi,
> >
> > I just setup a test ceph installation on 3 node Centos 6.5 .
> > two of the nodes are used for hosting osds and the third acts as mon .
> >
> > Please note I'm using LVM so had to set up the osd using the manual
> install
> > guide.
> >
> > --snip--
> > ceph -s
> > cluster 2929fa80-0841-4cb6-a133-90b2098fc802
> > health HEALTH_WARN 192 pgs stuck inactive; 192 pgs stuck unclean;
> > noup,nodown,noout flag(s) set
> > monmap e2: 3 mons at{ceph0=
> 10.0.12.220:6789/0,ceph1=10.0.12.221:6789/0,ceph2=10.0.12.222:6789/0
> > }, election epoch 46, quorum 0,1,2 ceph0,ceph1,ceph2
> > osdmap e21: 2 osds: 0 up, 0 in
> > flags noup,nodown,noout
>
>
> Do 'ceph osd unset noup' and they should start up. You likely also want
> to clear nodown and noout as well.
>
> sage
>
>
> > pgmap v22: 192 pgs, 3 pools, 0 bytes data, 0 objects
> > 0 kB used, 0 kB / 0 kB avail
> > 192 creating
> > --snip--
> >
> > osd tree
> >
> > --snip--
> > ceph osd tree
> > # idweight type name up/down reweight
> > -1 2 root default
> > -3 1 host ceph1
> > 0 1 osd.0 down0
> > -2 1 host ceph2
> > 1 1 osd.1 down0
> > --snip--
> >
> > --snip--
> > ceph daemon osd.0 status
> > { "cluster_fsid": "99babb8f-c880-4b32-a227-94aa483d4871",
> > "osd_fsid": "1ad28bde-c23c-44ba-a3b7-0fd3372e",
> > "whoami": 0,
> > "state": "booting",
> > "oldest_map": 1,
> > "newest_map": 21,
> > "num_pgs": 0}
> >
> > --snip--
> >
> > --snip--
> > ceph daemon osd.1 status
> > { "cluster_fsid": "99babb8f-c880-4b32-a227-94aa483d4871",
> > "osd_fsid": "becc3252-6977-47d6-87af-7b1337e591d8",
> > "whoami": 1,
> > "state": "booting",
> > "oldest_map": 1,
> > "newest_map": 21,
> > "num_pgs": 0}
> > --snip--
> >
> > # Cpus are idling
> >
> > # does anybody know what is wrong
> >
> > Thanks in advance
> >
> >
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] firefly osds stuck in state booting
On Sat, 26 Jul 2014, 10 minus wrote:
> Hi,
>
> I just setup a test ceph installation on 3 node Centos 6.5 .
> two of the nodes are used for hosting osds and the third acts as mon .
>
> Please note I'm using LVM so had to set up the osd using the manual install
> guide.
>
> --snip--
> ceph -s
> cluster 2929fa80-0841-4cb6-a133-90b2098fc802
> health HEALTH_WARN 192 pgs stuck inactive; 192 pgs stuck unclean;
> noup,nodown,noout flag(s) set
> monmap e2: 3 mons
> at{ceph0=10.0.12.220:6789/0,ceph1=10.0.12.221:6789/0,ceph2=10.0.12.222:6789/0
> }, election epoch 46, quorum 0,1,2 ceph0,ceph1,ceph2
> osdmap e21: 2 osds: 0 up, 0 in
> flags noup,nodown,noout
Do 'ceph osd unset noup' and they should start up. You likely also want
to clear nodown and noout as well.
sage
> pgmap v22: 192 pgs, 3 pools, 0 bytes data, 0 objects
> 0 kB used, 0 kB / 0 kB avail
> 192 creating
> --snip--
>
> osd tree
>
> --snip--
> ceph osd tree
> # id weight type name up/down reweight
> -1 2 root default
> -3 1 host ceph1
> 0 1 osd.0 down 0
> -2 1 host ceph2
> 1 1 osd.1 down 0
> --snip--
>
> --snip--
> ceph daemon osd.0 status
> { "cluster_fsid": "99babb8f-c880-4b32-a227-94aa483d4871",
> "osd_fsid": "1ad28bde-c23c-44ba-a3b7-0fd3372e",
> "whoami": 0,
> "state": "booting",
> "oldest_map": 1,
> "newest_map": 21,
> "num_pgs": 0}
>
> --snip--
>
> --snip--
> ceph daemon osd.1 status
> { "cluster_fsid": "99babb8f-c880-4b32-a227-94aa483d4871",
> "osd_fsid": "becc3252-6977-47d6-87af-7b1337e591d8",
> "whoami": 1,
> "state": "booting",
> "oldest_map": 1,
> "newest_map": 21,
> "num_pgs": 0}
> --snip--
>
> # Cpus are idling
>
> # does anybody know what is wrong
>
> Thanks in advance
>
> ___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] firefly osds stuck in state booting
Hi,
I just setup a test ceph installation on 3 node Centos 6.5 .
two of the nodes are used for hosting osds and the third acts as mon .
Please note I'm using LVM so had to set up the osd using the manual install
guide.
--snip--
ceph -s
cluster 2929fa80-0841-4cb6-a133-90b2098fc802
health HEALTH_WARN 192 pgs stuck inactive; 192 pgs stuck unclean;
noup,nodown,noout flag(s) set
monmap e2: 3 mons at {ceph0=
10.0.12.220:6789/0,ceph1=10.0.12.221:6789/0,ceph2=10.0.12.222:6789/0},
election epoch 46, quorum 0,1,2 ceph0,ceph1,ceph2
osdmap e21: 2 osds: 0 up, 0 in
flags noup,nodown,noout
pgmap v22: 192 pgs, 3 pools, 0 bytes data, 0 objects
0 kB used, 0 kB / 0 kB avail
192 creating
--snip--
osd tree
--snip--
ceph osd tree
# idweight type name up/down reweight
-1 2 root default
-3 1 host ceph1
0 1 osd.0 down0
-2 1 host ceph2
1 1 osd.1 down0
--snip--
--snip--
ceph daemon osd.0 status
{ "cluster_fsid": "99babb8f-c880-4b32-a227-94aa483d4871",
"osd_fsid": "1ad28bde-c23c-44ba-a3b7-0fd3372e",
"whoami": 0,
"state": "booting",
"oldest_map": 1,
"newest_map": 21,
"num_pgs": 0}
--snip--
--snip--
ceph daemon osd.1 status
{ "cluster_fsid": "99babb8f-c880-4b32-a227-94aa483d4871",
"osd_fsid": "becc3252-6977-47d6-87af-7b1337e591d8",
"whoami": 1,
"state": "booting",
"oldest_map": 1,
"newest_map": 21,
"num_pgs": 0}
--snip--
# Cpus are idling
# does anybody know what is wrong
Thanks in advance
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
