Re: [ceph-users] Orphaned entries in Crush map

2018-02-16 Thread David Turner 
First you stop the service, then make sure they're down, out, crush remove,
auth del, and finally osd rm.  You had it almost in the right order, but
you were down and outing them before you stopped them.  That would allow
them to mark themselves back up and in.  The down and out commands don't
need the 'osd.', just the ${n}.

In any case, by this point the cluster definitely believes them to be down,
out, and removed from the cluster.  I swear I remember having phantom
devices in my crush map like this before, but I thought it was because the
osd hadn't been rm'd from the cluster... which doesn't seem to be the case.

Does anyone else have any thoughts?

On Fri, Feb 16, 2018 at 4:22 PM Karsten Becker 
wrote:

> Here is what I did - bash history:
>
> >  1897  for n in 6 7 14 15 16 17 18 19 3 9 10 11 12 20; do ceph osd down
> osd.$n; done>  1920  for n in 6 7 14 15 16 17 18 19 3 9 10 11 12 20; do
> ceph osd out
> osd.$n; done
> >  1921  for n in 6 7 14 15 16 17 18 19 3 9 10 11 12 20; do ceph osd down
> osd.$n; done
> >  1923  for n in 6 7 14 15 16 17 18 19 3 9 10 11 12 20; do systemctl stop
> ceph-osd@$n.service; done
> >  1925  for n in 6 7 14 15 16 17 18 19 3 9 10 11 12 20; do ceph osd crush
> remove osd.${n}; done
> >  1926  for n in 6 7 14 15 16 17 18 19 3 9 10 11 12 20; do ceph auth del
> osd.${n}; done
> >  1927  for n in 6 7 14 15 16 17 18 19 3 9 10 11 12 20; do ceph osd rm
> ${n}; done
>
> I assume that I did the right steps...
>
>
>
>
>
> On 16.02.2018 21:56, David Turner wrote:
> > What is the output of `ceph osd stat`?  My guess is that they are still
> > considered to be part of the cluster and going through the process of
> > removing OSDs from your cluster is what you need to do.  In particular
> > `ceph osd rm 19`.
> >
> > On Fri, Feb 16, 2018 at 2:31 PM Karsten Becker
> > > wrote:
> >
> > Hi.
> >
> > during the reorgainzation of my cluster I removed some OSDs.
> Obviously
> > something went wrong for 2 of them, osd.19 and osd.20.
> >
> > If I get my current Crush map, decompile and edit them, I see 2
> > orphaned/stale entries for the former OSDs:
> >
> > > device 16 osd.16 class hdd
> > > device 17 osd.17 class hdd
> > > device 18 osd.18 class hdd
> > > device 19 device19
> > > device 20 device20
> > > device 21 osd.21 class hdd
> > > device 22 osd.22 class hdd
> > > device 23 osd.23 class hdd
> >
> > If I delete them from the Crush map (file), recompile it and set it
> > productive - they appear again... if I get the current map again and
> > decompile them, they are in again.
> >
> > So how to get rid of these entries?
> >
> > Best from Berlin/Germany
> > Karsten
> >
> > Ecologic Institut gemeinnuetzige GmbH
> > Pfalzburger Str. 43/44, D-10717 Berlin
> > Geschaeftsfuehrerin / Director: Dr. Camilla Bausch
> > Sitz der Gesellschaft / Registered Office: Berlin (Germany)
> > Registergericht / Court of Registration: Amtsgericht Berlin
> > (Charlottenburg), HRB 57947
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com 
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
>
> Ecologic Institut gemeinnuetzige GmbH
> Pfalzburger Str. 43/44, D-10717 Berlin
> Geschaeftsfuehrerin / Director: Dr. Camilla Bausch
> Sitz der Gesellschaft / Registered Office: Berlin (Germany)
> Registergericht / Court of Registration: Amtsgericht Berlin
> (Charlottenburg), HRB 57947
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Orphaned entries in Crush map

2018-02-16 Thread Karsten Becker 
Here is what I did - bash history:

>  1897  for n in 6 7 14 15 16 17 18 19 3 9 10 11 12 20; do ceph osd down 
> osd.$n; done>  1920  for n in 6 7 14 15 16 17 18 19 3 9 10 11 12 20; do ceph 
> osd out
osd.$n; done
>  1921  for n in 6 7 14 15 16 17 18 19 3 9 10 11 12 20; do ceph osd down 
> osd.$n; done
>  1923  for n in 6 7 14 15 16 17 18 19 3 9 10 11 12 20; do systemctl stop 
> ceph-osd@$n.service; done
>  1925  for n in 6 7 14 15 16 17 18 19 3 9 10 11 12 20; do ceph osd crush 
> remove osd.${n}; done
>  1926  for n in 6 7 14 15 16 17 18 19 3 9 10 11 12 20; do ceph auth del 
> osd.${n}; done
>  1927  for n in 6 7 14 15 16 17 18 19 3 9 10 11 12 20; do ceph osd rm ${n}; 
> done

I assume that I did the right steps...





On 16.02.2018 21:56, David Turner wrote:
> What is the output of `ceph osd stat`?  My guess is that they are still
> considered to be part of the cluster and going through the process of
> removing OSDs from your cluster is what you need to do.  In particular
> `ceph osd rm 19`.
> 
> On Fri, Feb 16, 2018 at 2:31 PM Karsten Becker
> > wrote:
> 
> Hi.
> 
> during the reorgainzation of my cluster I removed some OSDs. Obviously
> something went wrong for 2 of them, osd.19 and osd.20.
> 
> If I get my current Crush map, decompile and edit them, I see 2
> orphaned/stale entries for the former OSDs:
> 
> > device 16 osd.16 class hdd
> > device 17 osd.17 class hdd
> > device 18 osd.18 class hdd
> > device 19 device19
> > device 20 device20
> > device 21 osd.21 class hdd
> > device 22 osd.22 class hdd
> > device 23 osd.23 class hdd
> 
> If I delete them from the Crush map (file), recompile it and set it
> productive - they appear again... if I get the current map again and
> decompile them, they are in again.
> 
> So how to get rid of these entries?
> 
> Best from Berlin/Germany
> Karsten
> 
> Ecologic Institut gemeinnuetzige GmbH
> Pfalzburger Str. 43/44, D-10717 Berlin
> Geschaeftsfuehrerin / Director: Dr. Camilla Bausch
> Sitz der Gesellschaft / Registered Office: Berlin (Germany)
> Registergericht / Court of Registration: Amtsgericht Berlin
> (Charlottenburg), HRB 57947
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 


Ecologic Institut gemeinnuetzige GmbH
Pfalzburger Str. 43/44, D-10717 Berlin
Geschaeftsfuehrerin / Director: Dr. Camilla Bausch
Sitz der Gesellschaft / Registered Office: Berlin (Germany)
Registergericht / Court of Registration: Amtsgericht Berlin (Charlottenburg), 
HRB 57947
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Orphaned entries in Crush map

2018-02-16 Thread Karsten Becker 
Hi David.

So far everything else is fine.

> 46 osds: 46 up, 46 in; 1344 remapped pgs

And the rm gives:

> root@kong[/0]:~ # ceph osd rm 19
> osd.19 does not exist. 
> root@kong[/0]:~ # ceph osd rm 20
> osd.20 does not exist.

The "devices" do NOT show up in "ceph osd tree" or "ceph osd df"... just
in the map.

If I do NOT delete them out of the crush map, compile and set it active,
I get
> 2 osds exist in the crush map but not in the osdmap
In that case they also do NOT show up in "ceph osd tree" or "ceph osd df".

:-(





On 16.02.2018 21:56, David Turner wrote:
> What is the output of `ceph osd stat`?  My guess is that they are still
> considered to be part of the cluster and going through the process of
> removing OSDs from your cluster is what you need to do.  In particular
> `ceph osd rm 19`.
> 
> On Fri, Feb 16, 2018 at 2:31 PM Karsten Becker
> > wrote:
> 
> Hi.
> 
> during the reorgainzation of my cluster I removed some OSDs. Obviously
> something went wrong for 2 of them, osd.19 and osd.20.
> 
> If I get my current Crush map, decompile and edit them, I see 2
> orphaned/stale entries for the former OSDs:
> 
> > device 16 osd.16 class hdd
> > device 17 osd.17 class hdd
> > device 18 osd.18 class hdd
> > device 19 device19
> > device 20 device20
> > device 21 osd.21 class hdd
> > device 22 osd.22 class hdd
> > device 23 osd.23 class hdd
> 
> If I delete them from the Crush map (file), recompile it and set it
> productive - they appear again... if I get the current map again and
> decompile them, they are in again.
> 
> So how to get rid of these entries?
> 
> Best from Berlin/Germany
> Karsten
> 
> Ecologic Institut gemeinnuetzige GmbH
> Pfalzburger Str. 43/44, D-10717 Berlin
> Geschaeftsfuehrerin / Director: Dr. Camilla Bausch
> Sitz der Gesellschaft / Registered Office: Berlin (Germany)
> Registergericht / Court of Registration: Amtsgericht Berlin
> (Charlottenburg), HRB 57947
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

-- 

Karsten Becker
Head of Information Technology
Ecologic Institute

Tel: +49 30 86880-137
Website: http://ecologic.eu

Ecologic Institut gemeinnuetzige GmbH
Pfalzburger Str. 43/44, D-10717 Berlin
Geschaeftsfuehrerin / Director: Dr. Camilla Bausch
Sitz der Gesellschaft / Registered Office: Berlin (Germany)
Registergericht / Court of Registration: Amtsgericht Berlin (Charlottenburg), 
HRB 57947
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Orphaned entries in Crush map

2018-02-16 Thread David Turner 
What is the output of `ceph osd stat`?  My guess is that they are still
considered to be part of the cluster and going through the process of
removing OSDs from your cluster is what you need to do.  In particular
`ceph osd rm 19`.

On Fri, Feb 16, 2018 at 2:31 PM Karsten Becker 
wrote:

> Hi.
>
> during the reorgainzation of my cluster I removed some OSDs. Obviously
> something went wrong for 2 of them, osd.19 and osd.20.
>
> If I get my current Crush map, decompile and edit them, I see 2
> orphaned/stale entries for the former OSDs:
>
> > device 16 osd.16 class hdd
> > device 17 osd.17 class hdd
> > device 18 osd.18 class hdd
> > device 19 device19
> > device 20 device20
> > device 21 osd.21 class hdd
> > device 22 osd.22 class hdd
> > device 23 osd.23 class hdd
>
> If I delete them from the Crush map (file), recompile it and set it
> productive - they appear again... if I get the current map again and
> decompile them, they are in again.
>
> So how to get rid of these entries?
>
> Best from Berlin/Germany
> Karsten
>
> Ecologic Institut gemeinnuetzige GmbH
> Pfalzburger Str. 43/44, D-10717 Berlin
> Geschaeftsfuehrerin / Director: Dr. Camilla Bausch
> Sitz der Gesellschaft / Registered Office: Berlin (Germany)
> Registergericht / Court of Registration: Amtsgericht Berlin
> (Charlottenburg), HRB 57947
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Orphaned entries in Crush map

2018-02-16 Thread Karsten Becker 
Hi.

during the reorgainzation of my cluster I removed some OSDs. Obviously
something went wrong for 2 of them, osd.19 and osd.20.

If I get my current Crush map, decompile and edit them, I see 2
orphaned/stale entries for the former OSDs:

> device 16 osd.16 class hdd
> device 17 osd.17 class hdd
> device 18 osd.18 class hdd
> device 19 device19
> device 20 device20
> device 21 osd.21 class hdd
> device 22 osd.22 class hdd
> device 23 osd.23 class hdd

If I delete them from the Crush map (file), recompile it and set it
productive - they appear again... if I get the current map again and
decompile them, they are in again.

So how to get rid of these entries?

Best from Berlin/Germany
Karsten

Ecologic Institut gemeinnuetzige GmbH
Pfalzburger Str. 43/44, D-10717 Berlin
Geschaeftsfuehrerin / Director: Dr. Camilla Bausch
Sitz der Gesellschaft / Registered Office: Berlin (Germany)
Registergericht / Court of Registration: Amtsgericht Berlin (Charlottenburg), 
HRB 57947
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com