Re: [ceph-users] what is Implicated osds

[Satish Patel]

Thanks Brad,

This is what i found, issue was MTU I have set MTU 9000 on all my OSD
nodes and mon node but somehow it get reverted on mon node back to
1500. Because of mismatched MTU caused some strange communication
issue between osd and mon nodes.

After fixing MTU on mon, things started clearing up.

On Mon, Aug 20, 2018 at 7:11 PM, Brad Hubbard  wrote:
> On Tue, Aug 21, 2018 at 2:37 AM, Satish Patel  wrote:
>> Folks,
>>
>> Today i found ceph -s is really slow and just hanging for minute or 2
>> minute to give me output also same with "ceph osd tree" output,
>> command just hanging long time to give me output..
>>
>> This is what i am seeing output, one OSD down not sure why its down
>> and what is the relation with command running slow?
>>
>> I am also seeing what does that means? " 369 slow requests are blocked
>>> 32 sec. Implicated osds 0,2,3,4,5,6,7,8,9,11"
>
> This is just a hint that these are the osds you should look at in
> regard to the slow requests.
>
> What's common about the stale pgs, what pool do they belong too and
> what are the configuration details of that pool?
>
> Can you do a pg query on one of the stale pgs?
>
>>
>>
>> [root@ostack-infra-01-ceph-mon-container-692bea95 ~]# ceph -s
>>   cluster:
>> id: c369cdc9-35a2-467a-981d-46e3e1af8570
>> health: HEALTH_WARN
>> Reduced data availability: 53 pgs stale
>> 369 slow requests are blocked > 32 sec. Implicated osds
>> 0,2,3,4,5,6,7,8,9,11
>>
>>   services:
>> mon: 3 daemons, quorum
>> ostack-infra-02-ceph-mon-container-87f0ee0e,ostack-infra-01-ceph-mon-container-692bea95,ostack-infra-03-ceph-mon-container-a92c1c2a
>> mgr: ostack-infra-01-ceph-mon-container-692bea95(active),
>> standbys: ostack-infra-03-ceph-mon-container-a92c1c2a,
>> ostack-infra-02-ceph-mon-container-87f0ee0e
>> osd: 12 osds: 11 up, 11 in
>>
>>   data:
>> pools:   5 pools, 656 pgs
>> objects: 1461 objects, 11509 MB
>> usage:   43402 MB used, 5080 GB / 5122 GB avail
>> pgs: 603 active+clean
>>  53  stale+active+clean
>>
>>
>>
>> [root@ostack-infra-01-ceph-mon-container-692bea95 ~]# ceph osd tree
>> ID CLASS WEIGHT  TYPE NAMESTATUS REWEIGHT PRI-AFF
>> -1   5.45746 root default
>> -3   1.81915 host ceph-osd-01
>>  0   ssd 0.45479 osd.0up  1.0 1.0
>>  2   ssd 0.45479 osd.2up  1.0 1.0
>>  5   ssd 0.45479 osd.5up  1.0 1.0
>>  6   ssd 0.45479 osd.6up  1.0 1.0
>> -5   1.81915 host ceph-osd-02
>>  1   ssd 0.45479 osd.1  down0 1.0
>>  3   ssd 0.45479 osd.3up  1.0 1.0
>>  4   ssd 0.45479 osd.4up  1.0 1.0
>>  7   ssd 0.45479 osd.7up  1.0 1.0
>> -7   1.81915 host ceph-osd-03
>>  8   ssd 0.45479 osd.8up  1.0 1.0
>>  9   ssd 0.45479 osd.9up  1.0 1.0
>> 10   ssd 0.45479 osd.10   up  1.0 1.0
>> 11   ssd 0.45479 osd.11   up  1.0 1.0
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
> --
> Cheers,
> Brad
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ensure Hammer client compatibility

[Kees Meijs]

Hi Lincoln,

We're looking at (now existing) RBD support using KVM/QEMU, so this is 
an upgrade path.


Regards,
Kees

On 20-08-18 16:37, Lincoln Bryant wrote:

What interfaces do your Hammer clients need? If you're looking at
CephFS, we have had reasonable success moving our older clients (EL6)
to NFS Ganesha with the Ceph FSAL.



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Upgrade to Infernalis: OSDs crash all the time

[Kees Meijs]

Hi there,

A few hours ago I started the given OSD again and gave it weight 
1.0. Backfilling started and more PGs became active+clean.


After a while the same crashing behaviour started to act up so I stopped 
the backfilling.


Running with noout,nobackfill,norebalance,noscrub,nodeep-scrub flags now 
but at least it seems the cluster seems stable (fingers crossed...)


Possible plan of attack:

1. Stopping all Infernalis OSDs.
2. Remove Ceph Infernalis packages from OSD node.
3. Install Hammer packages.
4. Start the OSDs (or maybe the package installation does this already.)

Effectively this is an OSD downgrade. Is this supported or does Ceph 
"upgrade" data structures on disk as well?


Recap: this would imply going from Infernalis back to Hammer.

Any thoughts are more than welcome (maybe a completely different 
approach makes sense...) Meanwhile, I'll try to catch some sleep.


Thanks, thanks!

Best regards,
Kees

On 20-08-18 21:46, Kees Meijs wrote:


Other than restarting the "out" and stopped OSD for the time being 
(haven't tried that yet) I'm quite lost.




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] packages names for ubuntu/debian

[Alfredo Daniel Rezinovsky]

On 20/08/18 03:50, Bastiaan Visser wrote:

you should only use the 18.04 repo in 18.04, and remove the 16.04 repo.

use:
https://download.ceph.com/debian-luminous bionic main

- Bastiaan


Right. But if I came from a working 16.04 system upgraded to 18.04 the 
ceph (xenial) packages are already there and wont upgrade to beaver ones 
because the names means downgrade.



- Original Message -
From: "Alfredo Daniel Rezinovsky" 
To: "ceph-users" 
Sent: Sunday, August 19, 2018 10:15:00 PM
Subject: [ceph-users] packages names for ubuntu/debian

Last packages for ubuntu 16.04 are version 13.2.1-1xenial
while last packages for ubuntu 18.04 are 13.2.1-1bionic

I recently upgraded from ubuntu 16 to 18 and the ceph packages stayed in xenial 
because alphabetically xenial > bionic.

I had to set the piining to force the upgrade to bionic (Which was trated as a 
downgrade)

In Ubuntu maling lists they said those packages are "wrongly versioned"

I think the names should be 13.2.1-1ubuntu16.04-xenial and 
13.2.1.ubuntu18.04-bionic.



--
Alfredo Daniel Rezinovsky
Director de Tecnologías de Información y Comunicaciones
Facultad de Ingeniería - Universidad Nacional de Cuyo

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] what is Implicated osds

[Brad Hubbard]

On Tue, Aug 21, 2018 at 2:37 AM, Satish Patel  wrote:
> Folks,
>
> Today i found ceph -s is really slow and just hanging for minute or 2
> minute to give me output also same with "ceph osd tree" output,
> command just hanging long time to give me output..
>
> This is what i am seeing output, one OSD down not sure why its down
> and what is the relation with command running slow?
>
> I am also seeing what does that means? " 369 slow requests are blocked
>> 32 sec. Implicated osds 0,2,3,4,5,6,7,8,9,11"

This is just a hint that these are the osds you should look at in
regard to the slow requests.

What's common about the stale pgs, what pool do they belong too and
what are the configuration details of that pool?

Can you do a pg query on one of the stale pgs?

>
>
> [root@ostack-infra-01-ceph-mon-container-692bea95 ~]# ceph -s
>   cluster:
> id: c369cdc9-35a2-467a-981d-46e3e1af8570
> health: HEALTH_WARN
> Reduced data availability: 53 pgs stale
> 369 slow requests are blocked > 32 sec. Implicated osds
> 0,2,3,4,5,6,7,8,9,11
>
>   services:
> mon: 3 daemons, quorum
> ostack-infra-02-ceph-mon-container-87f0ee0e,ostack-infra-01-ceph-mon-container-692bea95,ostack-infra-03-ceph-mon-container-a92c1c2a
> mgr: ostack-infra-01-ceph-mon-container-692bea95(active),
> standbys: ostack-infra-03-ceph-mon-container-a92c1c2a,
> ostack-infra-02-ceph-mon-container-87f0ee0e
> osd: 12 osds: 11 up, 11 in
>
>   data:
> pools:   5 pools, 656 pgs
> objects: 1461 objects, 11509 MB
> usage:   43402 MB used, 5080 GB / 5122 GB avail
> pgs: 603 active+clean
>  53  stale+active+clean
>
>
>
> [root@ostack-infra-01-ceph-mon-container-692bea95 ~]# ceph osd tree
> ID CLASS WEIGHT  TYPE NAMESTATUS REWEIGHT PRI-AFF
> -1   5.45746 root default
> -3   1.81915 host ceph-osd-01
>  0   ssd 0.45479 osd.0up  1.0 1.0
>  2   ssd 0.45479 osd.2up  1.0 1.0
>  5   ssd 0.45479 osd.5up  1.0 1.0
>  6   ssd 0.45479 osd.6up  1.0 1.0
> -5   1.81915 host ceph-osd-02
>  1   ssd 0.45479 osd.1  down0 1.0
>  3   ssd 0.45479 osd.3up  1.0 1.0
>  4   ssd 0.45479 osd.4up  1.0 1.0
>  7   ssd 0.45479 osd.7up  1.0 1.0
> -7   1.81915 host ceph-osd-03
>  8   ssd 0.45479 osd.8up  1.0 1.0
>  9   ssd 0.45479 osd.9up  1.0 1.0
> 10   ssd 0.45479 osd.10   up  1.0 1.0
> 11   ssd 0.45479 osd.11   up  1.0 1.0
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
Cheers,
Brad
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph balancer: further optimizations?

[Stefan Priebe - Profihost AG]

Am 20.08.2018 um 22:38 schrieb Dan van der Ster:
> On Mon, Aug 20, 2018 at 10:19 PM Stefan Priebe - Profihost AG
>  wrote:
>>
>>
>> Am 20.08.2018 um 21:52 schrieb Sage Weil:
>>> On Mon, 20 Aug 2018, Stefan Priebe - Profihost AG wrote:
 Hello,

 since loic seems to have left ceph development and his wunderful crush
 optimization tool isn'T working anymore i'm trying to get a good
 distribution with the ceph balancer.

 Sadly it does not work as good as i want.

 # ceph osd df | sort -k8

 show 75 to 83% Usage which is 8% difference which is too much for me.
 I'm optimization by bytes.

 # ceph balancer eval
 current cluster score 0.005420 (lower is better)

 # ceph balancer eval $OPT_NAME
 plan spriebe_2018-08-20_19:36 final score 0.005456 (lower is better)

 I'm unable to optimize further ;-( Is there any chance to optimize
 further even in case of more rebelancing?
>>>
>>> The scoring that the balancer module is doing is currently a hybrid of pg
>>> count, bytes, and object count.  Picking a single metric might help a bit
>>> (as those 3 things are not always perfectly aligned).
>>
>> Hi,
>>
>> ok i found a bug in the balancer code which seems to be present in all
>> releases.
>>
>>  861 best_ws = next_ws
>>  862 best_ow = next_ow
>>
>>
>> should be:
>>
>>  861 best_ws = copy.deepcopy(next_ws)
>>  862 best_ow = copy.deepcopy(next_ow)
>>
>> otherwise it does not use the best but the last.
> 
> Interesting... does that change improve things?

It fixes the following (mgr debug output):
2018-08-20 22:33:46.078525 7f2fbc3b6700  0 mgr[balancer] Step result
score 0.001152 -> 0.001180, misplacing 0.000912
2018-08-20 22:33:46.078574 7f2fbc3b6700  0 mgr[balancer] Score got
worse, taking another step
2018-08-20 22:33:46.078770 7f2fbc3b6700  0 mgr[balancer] Balancing root
default (pools ['cephstor2']) by bytes
2018-08-20 22:33:46.156326 7f2fbc3b6700  0 mgr[balancer] Step result
score 0.001152 -> 0.001180, misplacing 0.000912
2018-08-20 22:33:46.156374 7f2fbc3b6700  0 mgr[balancer] Score got
worse, taking another step
2018-08-20 22:33:46.156581 7f2fbc3b6700  0 mgr[balancer] Balancing root
default (pools ['cephstor2']) by bytes
2018-08-20 22:33:46.233818 7f2fbc3b6700  0 mgr[balancer] Step result
score 0.001152 -> 0.001180, misplacing 0.000912
2018-08-20 22:33:46.233868 7f2fbc3b6700  0 mgr[balancer] Score got
worse, taking another step
2018-08-20 22:33:46.234043 7f2fbc3b6700  0 mgr[balancer] Balancing root
default (pools ['cephstor2']) by bytes
2018-08-20 22:33:46.313212 7f2fbc3b6700  0 mgr[balancer] Step result
score 0.001152 -> 0.001180, misplacing 0.000912
2018-08-20 22:33:46.313714 7f2fbc3b6700  0 mgr[balancer] Score got
worse, trying smaller step 0.000244
2018-08-20 22:33:46.313887 7f2fbc3b6700  0 mgr[balancer] Balancing root
default (pools ['cephstor2']) by bytes
2018-08-20 22:33:46.391586 7f2fbc3b6700  0 mgr[balancer] Step result
score 0.001152 -> 0.001152, misplacing 0.001141
2018-08-20 22:33:46.393374 7f2fbc3b6700  0 mgr[balancer] Balancing root
default (pools ['cephstor2']) by bytes
2018-08-20 22:33:46.473956 7f2fbc3b6700  0 mgr[balancer] Step result
score 0.001152 -> 0.001180, misplacing 0.000912
2018-08-20 22:33:46.474001 7f2fbc3b6700  0 mgr[balancer] Score got
worse, taking another step
2018-08-20 22:33:46.474046 7f2fbc3b6700  0 mgr[balancer] Success, score
0.001155 -> 0.001152

BUT:
# ceph balancer eval myplan
plan myplan final score 0.001180 (lower is better)

So the final plan does NOT contain the expected optimization. The
deepcopy fixes it.

After:
# ceph balancer eval myplan
plan myplan final score 0.001152 (lower is better)

> 
> Also, if most of your data is in one pool you can try ceph balancer
> eval 

Already tried this doesn't help much.

Greets,
Stefan


> -- dan
> 
>>
>> I'm also using this one:
>> https://github.com/ceph/ceph/pull/20665/commits/c161a74ad6cf006cd9b33b40fd7705b67c170615
>>
>> to optimize by bytes only.
>>
>> Greets,
>> Stefan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph balancer: further optimizations?

[David Turner]

I didn't ask how many PGs per OSD, I asked how large are your PGs in
comparison to your OSDs.  For instance my primary data pool in my home
cluster has 10914GB of data in it and has 256 PGs.  That means that each PG
accounts for 42GB of data.  I'm using 5TB disks in this cluster.  Each PG
on an OSD accounts for 0.8% of the total disk capacity on each OSD.  With
that % size per PG, I can get a really good balance in that cluster since
even if an OSD had 5 more PGs than another, it's only different by 4% total.

Now let's change the amount of PGs in my cluster.  The same amount of data
and the same size of disk, but only 32 PGs in the pool.  Each PG in that
pool takes up 6.8% of an OSD's available space.  Now even if I only have 2
more PGs on one osd than another, it's off by 13.4%.

On Mon, Aug 20, 2018 at 4:29 PM Stefan Priebe - Profihost AG <
s.pri...@profihost.ag> wrote:

>
> Am 20.08.2018 um 22:13 schrieb David Turner:
> > You might just have too much data per PG.  If a single PG can account
> > for 4% of your OSD, then 9% difference in used space on your OSDs is
> > caused by an OSD having only 2 more PGs than another OSD.  If you do
> > have very large PGs, increasing your PG count in those pools should
> > improve your data distribution.
>
> 4384 pgs
> 91 osds
> 3x replication
>
> 4384 / 91 * 3 => ~ 150 pgs which seems to be even too much - at least
> the doc says around 100 pgs / osd.
>
> >
> > On Mon, Aug 20, 2018 at 3:59 PM Sage Weil  > > wrote:
> >
> > On Mon, 20 Aug 2018, Stefan Priebe - Profihost AG wrote:
> > > Hello,
> > >
> > > since loic seems to have left ceph development and his wunderful
> crush
> > > optimization tool isn'T working anymore i'm trying to get a good
> > > distribution with the ceph balancer.
> > >
> > > Sadly it does not work as good as i want.
> > >
> > > # ceph osd df | sort -k8
> > >
> > > show 75 to 83% Usage which is 8% difference which is too much for
> me.
> > > I'm optimization by bytes.
> > >
> > > # ceph balancer eval
> > > current cluster score 0.005420 (lower is better)
> > >
> > > # ceph balancer eval $OPT_NAME
> > > plan spriebe_2018-08-20_19:36 final score 0.005456 (lower is
> better)
> > >
> > > I'm unable to optimize further ;-( Is there any chance to optimize
> > > further even in case of more rebelancing?
> >
> > The scoring that the balancer module is doing is currently a hybrid
> > of pg
> > count, bytes, and object count.  Picking a single metric might help
> > a bit
> > (as those 3 things are not always perfectly aligned).
> >
> > s
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com 
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph balancer: further optimizations?

[Stefan Priebe - Profihost AG]

Am 20.08.2018 um 22:13 schrieb David Turner:
> You might just have too much data per PG.  If a single PG can account
> for 4% of your OSD, then 9% difference in used space on your OSDs is
> caused by an OSD having only 2 more PGs than another OSD.  If you do
> have very large PGs, increasing your PG count in those pools should
> improve your data distribution.

4384 pgs
91 osds
3x replication

4384 / 91 * 3 => ~ 150 pgs which seems to be even too much - at least
the doc says around 100 pgs / osd.

> 
> On Mon, Aug 20, 2018 at 3:59 PM Sage Weil  > wrote:
> 
> On Mon, 20 Aug 2018, Stefan Priebe - Profihost AG wrote:
> > Hello,
> >
> > since loic seems to have left ceph development and his wunderful crush
> > optimization tool isn'T working anymore i'm trying to get a good
> > distribution with the ceph balancer.
> >
> > Sadly it does not work as good as i want.
> >
> > # ceph osd df | sort -k8
> >
> > show 75 to 83% Usage which is 8% difference which is too much for me.
> > I'm optimization by bytes.
> >
> > # ceph balancer eval
> > current cluster score 0.005420 (lower is better)
> >
> > # ceph balancer eval $OPT_NAME
> > plan spriebe_2018-08-20_19:36 final score 0.005456 (lower is better)
> >
> > I'm unable to optimize further ;-( Is there any chance to optimize
> > further even in case of more rebelancing?
> 
> The scoring that the balancer module is doing is currently a hybrid
> of pg
> count, bytes, and object count.  Picking a single metric might help
> a bit
> (as those 3 things are not always perfectly aligned).
> 
> s
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph balancer: further optimizations?

[Stefan Priebe - Profihost AG]

Am 20.08.2018 um 21:52 schrieb Sage Weil:
> On Mon, 20 Aug 2018, Stefan Priebe - Profihost AG wrote:
>> Hello,
>>
>> since loic seems to have left ceph development and his wunderful crush
>> optimization tool isn'T working anymore i'm trying to get a good
>> distribution with the ceph balancer.
>>
>> Sadly it does not work as good as i want.
>>
>> # ceph osd df | sort -k8
>>
>> show 75 to 83% Usage which is 8% difference which is too much for me.
>> I'm optimization by bytes.
>>
>> # ceph balancer eval
>> current cluster score 0.005420 (lower is better)
>>
>> # ceph balancer eval $OPT_NAME
>> plan spriebe_2018-08-20_19:36 final score 0.005456 (lower is better)
>>
>> I'm unable to optimize further ;-( Is there any chance to optimize
>> further even in case of more rebelancing?
> 
> The scoring that the balancer module is doing is currently a hybrid of pg 
> count, bytes, and object count.  Picking a single metric might help a bit 
> (as those 3 things are not always perfectly aligned).

Hi,

ok i found a bug in the balancer code which seems to be present in all
releases.

 861 best_ws = next_ws
 862 best_ow = next_ow


should be:

 861 best_ws = copy.deepcopy(next_ws)
 862 best_ow = copy.deepcopy(next_ow)

otherwise it does not use the best but the last.

I'm also using this one:
https://github.com/ceph/ceph/pull/20665/commits/c161a74ad6cf006cd9b33b40fd7705b67c170615

to optimize by bytes only.

Greets,
Stefan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph balancer: further optimizations?

[David Turner]

You might just have too much data per PG.  If a single PG can account for
4% of your OSD, then 9% difference in used space on your OSDs is caused by
an OSD having only 2 more PGs than another OSD.  If you do have very large
PGs, increasing your PG count in those pools should improve your data
distribution.

On Mon, Aug 20, 2018 at 3:59 PM Sage Weil  wrote:

> On Mon, 20 Aug 2018, Stefan Priebe - Profihost AG wrote:
> > Hello,
> >
> > since loic seems to have left ceph development and his wunderful crush
> > optimization tool isn'T working anymore i'm trying to get a good
> > distribution with the ceph balancer.
> >
> > Sadly it does not work as good as i want.
> >
> > # ceph osd df | sort -k8
> >
> > show 75 to 83% Usage which is 8% difference which is too much for me.
> > I'm optimization by bytes.
> >
> > # ceph balancer eval
> > current cluster score 0.005420 (lower is better)
> >
> > # ceph balancer eval $OPT_NAME
> > plan spriebe_2018-08-20_19:36 final score 0.005456 (lower is better)
> >
> > I'm unable to optimize further ;-( Is there any chance to optimize
> > further even in case of more rebelancing?
>
> The scoring that the balancer module is doing is currently a hybrid of pg
> count, bytes, and object count.  Picking a single metric might help a bit
> (as those 3 things are not always perfectly aligned).
>
> s
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs client version in RedHat/CentOS 7.5

[Dan van der Ster]

On Mon, Aug 20, 2018 at 5:37 PM Ilya Dryomov  wrote:
>
> On Mon, Aug 20, 2018 at 4:52 PM Dietmar Rieder
>  wrote:
> >
> > Hi Cephers,
> >
> >
> > I wonder if the cephfs client in RedHat/CentOS 7.5 will be updated to
> > luminous?
> > As far as I see there is some luminous related stuff that was
> > backported, however,
> > the "ceph features" command just reports "jewel" as release of my cephfs
> > clients running CentOS 7.5 (kernel 3.10.0-862.11.6.el7.x86_64)
> >
> >
> > {
> > "mon": {
> > "group": {
> > "features": "0x3ffddff8eea4fffb",
> > "release": "luminous",
> > "num": 3
> > }
> > },
> > "mds": {
> > "group": {
> > "features": "0x3ffddff8eea4fffb",
> > "release": "luminous",
> > "num": 3
> > }
> > },
> > "osd": {
> > "group": {
> > "features": "0x3ffddff8eea4fffb",
> > "release": "luminous",
> > "num": 240
> > }
> > },
> > "client": {
> > "group": {
> > "features": "0x7010fb86aa42ada",
> > "release": "jewel",
> > "num": 23
> > },
> > "group": {
> > "features": "0x3ffddff8eea4fffb",
> > "release": "luminous",
> > "num": 4
> > }
> > }
> > }
> >
> >
> > This prevents me to run ceph balancer using the upmap mode.
> >
> >
> > Any idea?
>
> Hi Dietmar,
>
> All luminous features are supported in RedHat/CentOS 7.5, but it shows
> up as jewel due to a technicality.

Except rados namespaces, right? Manila CephFS shares are not yet
mountable with 7.5.

Cheers, Dan


-- dan

> Just do
>
>   $ ceph osd set-require-min-compat-client luminous --yes-i-really-mean-it
>
> to override the safety check.
>
> See https://www.spinics.net/lists/ceph-users/msg45071.html for details.
> It references an upstream kernel, but both the problem and the solution
> are the same.
>
> Thanks,
>
> Ilya
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Upgrade to Infernalis: OSDs crash all the time

[Kees Meijs]

Hi again,

I'm starting to feel really unlucky here...

At the moment, the situation is "sort of okay":


    1387 active+clean
  11 active+clean+inconsistent
   7 active+recovery_wait+degraded
   1 active+recovery_wait+undersized+degraded+remapped
   1 active+undersized+degraded+remapped+wait_backfill
   1 
active+undersized+degraded+remapped+inconsistent+backfilling


To ensure nothing is in the way, I disabled both scrubbing and deep 
scrubbing for the time being.


However, random OSDs (still on Hammer) constantly crash giving the error 
as mentioned earlier (osd/ReplicatedPG.cc: 10115: FAILED assert(r >= 0)).


It felt like they started crashing when hitting the PG currently 
backfilling, so I set the nobackfill flag.


For now, the crashing seems to have stopped. However, the cluster seems 
slow at the moment when trying to access the given PG via KVM/QEMU (RBD).


Recap:

 * All monitors run Infernalis.
 * One OSD node runs Infernalis.
 * All other OSD nodes run Hammer.
 * One OSD on Infernalis is set to "out" and is stopped. This OSD
   seemed to contain one inconsistent PG.
 * Backfilling started.
 * After hours and hours of backfilling, OSDs started to crash.

Other than restarting the "out" and stopped OSD for the time being 
(haven't tried that yet) I'm quite lost.


Hopefully someone has some pointers for me.

Regards,
Kees

On 20-08-18 13:23, Kees Meijs wrote:

The given PG is back online, phew...

Meanwhile, some OSDs still on Hammer seem to crash with errors alike:


2018-08-20 13:06:33.819569 7f8962b2f700 -1 osd/ReplicatedPG.cc: In
function 'void ReplicatedPG::scan_range(int, int,
PG::BackfillInterval*, ThreadPool::TPHandle&)' thread 7f8962b2f700
time 2018-08-20 13:06:33.709922
osd/ReplicatedPG.cc: 10115: FAILED assert(r >= 0)

Restarting the OSDs seems to work.

K.

On 20-08-18 13:14, Kees Meijs wrote:

Bad news: I've got a PG stuck in down+peering now.


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] missing dependecy in ubuntu packages

[John Spray]

On Mon, Aug 20, 2018 at 6:50 PM Alfredo Daniel Rezinovsky
 wrote:
>
>
>
> On 20/08/18 06:44, John Spray wrote:
> > On Sun, Aug 19, 2018 at 9:21 PM Alfredo Daniel Rezinovsky
> >  wrote:
> >> both in ubuntu 16.04 and 18.04 ceph-mgr fail to starts when package
> >> python-routes is not installed
> > I guess you mean that the dashboard doesn't work, as opposed to the
> > whole ceph-mgr process not starting?  If it's the latter that's
> > worrying...
> Yes, is the dashboard, not the whole mgr
> >
> > As far as I can tell, the dashboard doesn't actually use the routes
> > package itself.  I wonder if there is another dependency that uses
> > routes, and that dependency's packaging has a missing dependency on
> > routes?  If you look in the ceph-mgr log you should see a backtrace
> > somewhere that would show which piece of code is trying to use the
> > routes module.
> the ceph status shows
>
> "Module 'dashboard' has failed: No module named routes"
>
> and installing python-routes fixes it.

Right, that's the dashboard taking the first line of a python
exception and exposing it as a health message.  However, the actual
exception may have originated in some other module that the dashboard
is using, rather than from the dashboard code itself.  So if you can
search your ceph-mgr log to find the backtrace (just search for the
first instance of "No module named routes"), then hopefully we can see
which piece of code is at fault.

Cheers,
John

>
>
> >
> > John
> >
> >> Some python packages are listed as dependencies for ceph-mgr but
> >> python-routes is missing and must be installed manually for ceph-mgr to
> >> work.
> >>
> >> --
> >> Alfredo Daniel Rezinovsky
> >> Director de Tecnologías de Información y Comunicaciones
> >> Facultad de Ingeniería - Universidad Nacional de Cuyo
> >>
> >> ___
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> --
> Alfredo Daniel Rezinovsky
> Director de Tecnologías de Información y Comunicaciones
> Facultad de Ingeniería - Universidad Nacional de Cuyo
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph balancer: further optimizations?

[Stefan Priebe - Profihost AG]

Hello,

since loic seems to have left ceph development and his wunderful crush
optimization tool isn'T working anymore i'm trying to get a good
distribution with the ceph balancer.

Sadly it does not work as good as i want.

# ceph osd df | sort -k8

show 75 to 83% Usage which is 8% difference which is too much for me.
I'm optimization by bytes.

# ceph balancer eval
current cluster score 0.005420 (lower is better)

# ceph balancer eval $OPT_NAME
plan spriebe_2018-08-20_19:36 final score 0.005456 (lower is better)

I'm unable to optimize further ;-( Is there any chance to optimize
further even in case of more rebelancing?

Greets,
Stefan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] missing dependecy in ubuntu packages

[Alfredo Daniel Rezinovsky]

On 20/08/18 06:44, John Spray wrote:

On Sun, Aug 19, 2018 at 9:21 PM Alfredo Daniel Rezinovsky
 wrote:

both in ubuntu 16.04 and 18.04 ceph-mgr fail to starts when package
python-routes is not installed

I guess you mean that the dashboard doesn't work, as opposed to the
whole ceph-mgr process not starting?  If it's the latter that's
worrying...

Yes, is the dashboard, not the whole mgr


As far as I can tell, the dashboard doesn't actually use the routes
package itself.  I wonder if there is another dependency that uses
routes, and that dependency's packaging has a missing dependency on
routes?  If you look in the ceph-mgr log you should see a backtrace
somewhere that would show which piece of code is trying to use the
routes module.

the ceph status shows

"Module 'dashboard' has failed: No module named routes"

and installing python-routes fixes it.




John


Some python packages are listed as dependencies for ceph-mgr but
python-routes is missing and must be installed manually for ceph-mgr to
work.

--
Alfredo Daniel Rezinovsky
Director de Tecnologías de Información y Comunicaciones
Facultad de Ingeniería - Universidad Nacional de Cuyo

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


--
Alfredo Daniel Rezinovsky
Director de Tecnologías de Información y Comunicaciones
Facultad de Ingeniería - Universidad Nacional de Cuyo

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] failing to respond to cache pressure

[Zhenshi Zhou]

Hi Eugen,
I think it does have positive effect on the messages. Cause I get fewer
messages than before.

Eugen Block  于2018年8月20日周一 下午9:29写道：

> Update: we are getting these messages again.
>
> So the search continues...
>
>
> Zitat von Eugen Block :
>
> > Hi,
> >
> > Depending on your kernel (memory leaks with CephFS) increasing the
> > mds_cache_memory_limit could be of help. What is your current
> > setting now?
> >
> > ceph:~ # ceph daemon mds. config show | grep mds_cache_memory_limit
> >
> > We had these messages for months, almost every day.
> > It would occur when hourly backup jobs ran and the MDS had to serve
> > an additional client (searching the whole CephFS for changes)
> > besides the existing CephFS clients. First we updated all clients to
> > a more recent kernel version, but the warnings didn't stop. Then we
> > doubled the cache size from 2 GB to 4 GB last week and since then I
> > haven't seen this warning again (for now).
> >
> > Try playing with the cache size to find a setting fitting your
> > needs, but don't forget to monitor your MDS in case something goes
> > wrong.
> >
> > Regards,
> > Eugen
> >
> >
> > Zitat von Wido den Hollander :
> >
> >> On 08/13/2018 01:22 PM, Zhenshi Zhou wrote:
> >>> Hi,
> >>> Recently, the cluster runs healthy, but I get warning messages
> everyday:
> >>>
> >>
> >> Which version of Ceph? Which version of clients?
> >>
> >> Can you post:
> >>
> >> $ ceph versions
> >> $ ceph features
> >> $ ceph fs status
> >>
> >> Wido
> >>
> >>> 2018-08-13 17:39:23.682213 [INF]  Cluster is now healthy
> >>> 2018-08-13 17:39:23.682144 [INF]  Health check cleared:
> >>> MDS_CLIENT_RECALL (was: 6 clients failing to respond to cache pressure)
> >>> 2018-08-13 17:39:23.052022 [INF]  MDS health message cleared (mds.0):
> >>> Client docker38:docker failing to respond to cache pressure
> >>> 2018-08-13 17:39:23.051979 [INF]  MDS health message cleared (mds.0):
> >>> Client docker73:docker failing to respond to cache pressure
> >>> 2018-08-13 17:39:23.051934 [INF]  MDS health message cleared (mds.0):
> >>> Client docker74:docker failing to respond to cache pressure
> >>> 2018-08-13 17:39:23.051853 [INF]  MDS health message cleared (mds.0):
> >>> Client docker75:docker failing to respond to cache pressure
> >>> 2018-08-13 17:39:23.051815 [INF]  MDS health message cleared (mds.0):
> >>> Client docker27:docker failing to respond to cache pressure
> >>> 2018-08-13 17:39:23.051753 [INF]  MDS health message cleared (mds.0):
> >>> Client docker27 failing to respond to cache pressure
> >>> 2018-08-13 17:38:11.100331 [WRN]  Health check update: 6 clients
> failing
> >>> to respond to cache pressure (MDS_CLIENT_RECALL)
> >>> 2018-08-13 17:37:39.570014 [WRN]  Health check update: 5 clients
> failing
> >>> to respond to cache pressure (MDS_CLIENT_RECALL)
> >>> 2018-08-13 17:37:31.099418 [WRN]  Health check update: 3 clients
> failing
> >>> to respond to cache pressure (MDS_CLIENT_RECALL)
> >>> 2018-08-13 17:36:34.564345 [WRN]  Health check update: 1 clients
> failing
> >>> to respond to cache pressure (MDS_CLIENT_RECALL)
> >>> 2018-08-13 17:36:27.121891 [WRN]  Health check update: 3 clients
> failing
> >>> to respond to cache pressure (MDS_CLIENT_RECALL)
> >>> 2018-08-13 17:36:11.967531 [WRN]  Health check update: 5 clients
> failing
> >>> to respond to cache pressure (MDS_CLIENT_RECALL)
> >>> 2018-08-13 17:35:59.870055 [WRN]  Health check update: 6 clients
> failing
> >>> to respond to cache pressure (MDS_CLIENT_RECALL)
> >>> 2018-08-13 17:35:47.787323 [WRN]  Health check update: 3 clients
> failing
> >>> to respond to cache pressure (MDS_CLIENT_RECALL)
> >>> 2018-08-13 17:34:59.435933 [WRN]  Health check failed: 1 clients
> failing
> >>> to respond to cache pressure (MDS_CLIENT_RECALL)
> >>> 2018-08-13 17:34:59.045510 [WRN]  MDS health message (mds.0): Client
> >>> docker75:docker failing to respond to cache pressure
> >>>
> >>> How can I fix it?
> >>>
> >>>
> >>> ___
> >>> ceph-users mailing list
> >>> ceph-users@lists.ceph.com
> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>>
> >> ___
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] QEMU/Libvirt + librbd issue using Luminous 12.2.7

[Andre Goree]

This issue first started while using Luminous 12.2.5, I upgraded to 12.2.7 and 
it's still present.  This issue is _not_ present in 12.2.4.

With Ceph 12.2.4, using QEMU/KVM + Libvirt, I'm able to mount an rbd image 
using the following syntax and populated xml:

'virsh attach-device $vm foo.xml --persistent'

xml contents:

 
  



   
   
 
   
   
   


I receive this error:
~# virsh attach-device $vm foo.xml --persistent
error: Failed to attach device from foo.xml
error: internal error: unable to execute QEMU command 'device_add': Property 
'scsi-hd.drive' can't find value 'drive-scsi0-0-0-1'

I've tried different things with the XML, but nothing seems to work, always 
failing with the above error.  This does _not_ happen with our cluster running 
12.2.4, the same exact command with a cluster using an identical configuration 
(for all intents and purposes).

Any thoughts?  Hard to believe I'm the only one to hit this if it's indeed a 
bug, but I haven't found anyone else having the issue through interweb searches.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Removing all rados objects based on a prefix

[Wido den Hollander]

On 08/20/2018 05:20 PM, David Turner wrote:
> The general talk about the rados cleanup command is to clean things up
> after benchmarking.  Could this command also be used for deleting an old
> RGW bucket or an RBD.  For instance, a bucket with a prefix of
> `25ff9eff-058b-41e3-8724-cfffecb979c0.9709451.1` such that all objects
> in the default.rgw.buckets.data pool for that bucket start with that
> string.  Could I run [1] this command to clean all of those up?  Listing
> the full pool contents and grepping out for that string returns 100M
> objects and every way I've come up with to iterate over that list will
> take us about a month to get through it.  I would think this has a
> decent chance to work, except for the description of the [2] cleanup
> option from the rados man page.
> 
> Perhaps I'm also barking up the wrong tree.  Does anyone have a better
> way to delete large RBDs or buckets?
> 

Nope, you can't filter on prefixes of objects. You'll have to do a
listing and filter on the output.. That's a very long list.

Wido

> 
> [1] rados -p default.rgw.buckets.data cleanup --prefix
> 25ff9eff-058b-41e3-8724-cfffecb979c0.9709451.1
> 
> [2] cleanup [ --run-name run_name ] [ --prefix prefix ]
>         Clean up a previous benchmark operation.  Note: the default
> run-name is "benchmark_last_metadata"
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] what is Implicated osds

[Satish Patel]

Folks,

Today i found ceph -s is really slow and just hanging for minute or 2
minute to give me output also same with "ceph osd tree" output,
command just hanging long time to give me output..

This is what i am seeing output, one OSD down not sure why its down
and what is the relation with command running slow?

I am also seeing what does that means? " 369 slow requests are blocked
> 32 sec. Implicated osds 0,2,3,4,5,6,7,8,9,11"


[root@ostack-infra-01-ceph-mon-container-692bea95 ~]# ceph -s
  cluster:
id: c369cdc9-35a2-467a-981d-46e3e1af8570
health: HEALTH_WARN
Reduced data availability: 53 pgs stale
369 slow requests are blocked > 32 sec. Implicated osds
0,2,3,4,5,6,7,8,9,11

  services:
mon: 3 daemons, quorum
ostack-infra-02-ceph-mon-container-87f0ee0e,ostack-infra-01-ceph-mon-container-692bea95,ostack-infra-03-ceph-mon-container-a92c1c2a
mgr: ostack-infra-01-ceph-mon-container-692bea95(active),
standbys: ostack-infra-03-ceph-mon-container-a92c1c2a,
ostack-infra-02-ceph-mon-container-87f0ee0e
osd: 12 osds: 11 up, 11 in

  data:
pools:   5 pools, 656 pgs
objects: 1461 objects, 11509 MB
usage:   43402 MB used, 5080 GB / 5122 GB avail
pgs: 603 active+clean
 53  stale+active+clean



[root@ostack-infra-01-ceph-mon-container-692bea95 ~]# ceph osd tree
ID CLASS WEIGHT  TYPE NAMESTATUS REWEIGHT PRI-AFF
-1   5.45746 root default
-3   1.81915 host ceph-osd-01
 0   ssd 0.45479 osd.0up  1.0 1.0
 2   ssd 0.45479 osd.2up  1.0 1.0
 5   ssd 0.45479 osd.5up  1.0 1.0
 6   ssd 0.45479 osd.6up  1.0 1.0
-5   1.81915 host ceph-osd-02
 1   ssd 0.45479 osd.1  down0 1.0
 3   ssd 0.45479 osd.3up  1.0 1.0
 4   ssd 0.45479 osd.4up  1.0 1.0
 7   ssd 0.45479 osd.7up  1.0 1.0
-7   1.81915 host ceph-osd-03
 8   ssd 0.45479 osd.8up  1.0 1.0
 9   ssd 0.45479 osd.9up  1.0 1.0
10   ssd 0.45479 osd.10   up  1.0 1.0
11   ssd 0.45479 osd.11   up  1.0 1.0
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Mimic osd fails to start.

[Daznis]

Hello,


Medic shows everything fine. Whole cluster is on the latest mimic
version. It was updated to mimic when stable version of mimic was
release and recently it was updated to "ceph version 13.2.1
(5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable)". For some
reason one mgr service is running, but it's not connected to the
cluster.

Versions output:

{
"mon": {
"ceph version 13.2.1
(5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable)": 3
},
"mgr": {
"ceph version 13.2.1
(5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable)": 2
},
"osd": {
"ceph version 13.2.1
(5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable)": 47
},
"mds": {},
"overall": {
"ceph version 13.2.1
(5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable)": 52
}
}

Medic output:
===  Starting remote check session  
Version: 1.0.4Cluster Name: "ceph"
Total hosts: [10]
OSDs:5MONs:3 Clients:0
MDSs:0RGWs:0 MGRs:   2



-- managers --
 mon03
 mon02
 mon01

 osds 
 node03
 node02
 node01
 node05
 node04

 mons 
 mon01
 mon03
 mon02

107 passed, on 11 hosts
On Mon, Aug 20, 2018 at 6:13 PM Alfredo Deza  wrote:
>
> On Mon, Aug 20, 2018 at 10:23 AM, Daznis  wrote:
> > Hello,
> >
> > It appears that something is horribly wrong with the cluster itself. I
> > can't create or add any new osds to it at all.
>
> Have you added new monitors? Or replaced monitors? I would check that
> all your versions match, something seems to be expecting different
> versions.
>
> The "Invalid argument" problem is a common thing we see when that happens.
>
> Something that might help a bit here is if you run ceph-medic against
> your cluster:
>
> http://docs.ceph.com/ceph-medic/master/
>
>
>
> > On Mon, Aug 20, 2018 at 11:04 AM Daznis  wrote:
> >>
> >> Hello,
> >>
> >>
> >> Zapping the journal didn't help. I tried to create the journal after
> >> zapping it. Also failed. I'm not really sure why this happens.
> >>
> >> Looking at the monitor logs with 20/20 debug I'm seeing these errors:
> >>
> >> 2018-08-20 08:57:58.753 7f9d85934700  0 mon.mon02@1(peon) e4
> >> handle_command mon_command({"prefix": "osd crush set-device-class",
> >> "class": "ssd", "ids": ["48"]} v 0) v1
> >> 2018-08-20 08:57:58.753 7f9d85934700 20 is_capable service=osd
> >> command=osd crush set-device-class read write on cap allow profile osd
> >> 2018-08-20 08:57:58.753 7f9d85934700 20  allow so far , doing grant
> >> allow profile osd
> >> 2018-08-20 08:57:58.753 7f9d85934700 20  match
> >> 2018-08-20 08:57:58.753 7f9d85934700 10 mon.mon02@1(peon) e4
> >> _allowed_command capable
> >> 2018-08-20 08:57:58.753 7f9d85934700  0 log_channel(audit) log [INF] :
> >> from='osd.48 10.24.52.17:6800/153683' entity='osd.48' cmd=[{"prefix":
> >> "osd crush set-device-class", "class": "ssd", "ids": ["48"]}]:
> >> dispatch
> >> 2018-08-20 08:57:58.753 7f9d85934700 10 mon.mon02@1(peon).osd e46327
> >> preprocess_query mon_command({"prefix": "osd crush set-device-class",
> >> "class": "ssd", "ids": ["48"]} v 0) v1 from osd.48
> >> 10.24.52.17:6800/153683
> >> 2018-08-20 08:57:58.753 7f9d85934700 10 mon.mon02@1(peon) e4
> >> forward_request 4 request mon_command({"prefix": "osd crush
> >> set-device-class", "class": "ssd", "ids": ["48"]} v 0) v1 features
> >> 4611087854031142907
> >> 2018-08-20 08:57:58.753 7f9d85934700 20 mon.mon02@1(peon) e4
> >> _ms_dispatch existing session 0x55b4ec482a80 for mon.1
> >> 10.24.52.11:6789/0
> >> 2018-08-20 08:57:58.753 7f9d85934700 20 mon.mon02@1(peon) e4  caps allow *
> >> 2018-08-20 08:57:58.753 7f9d85934700 10 mon.mon02@1(peon).log
> >> v10758065 preprocess_query log(1 entries from seq 4 at 2018-08-20
> >> 08:57:58.755306) v1 from mon.1 10.24.52.11:6789/0
> >> 2018-08-20 08:57:58.753 7f9d85934700 10 mon.mon02@1(peon).log
> >> v10758065 preprocess_log log(1 entries from seq 4 at 2018-08-20
> >> 08:57:58.755306) v1 from mon.1
> >> 2018-08-20 08:57:58.753 7f9d85934700 20 is_capable service=log
> >> command= write on cap allow *
> >> 2018-08-20 08:57:58.753 7f9d85934700 20  allow so far , doing grant allow *
> >> 2018-08-20 08:57:58.753 7f9d85934700 20  allow all
> >> 2018-08-20 08:57:58.753 7f9d85934700 10 mon.mon02@1(peon) e4
> >> forward_request 5 request log(1 entries from seq 4 at 2018-08-20
> >> 08:57:58.755306) v1 features 4611087854031142907
> >> 2018-08-20 08:57:58.754 7f9d85934700 20 mon.mon02@1(peon) e4
> >> _ms_dispatch existing session 0x55b4ec4828c0 for mon.0
> >> 10.24.52.10:6789/0
> >> 2018-08-20 08:57:58.754 7f9d85934700 20 mon.mon02@1(peon) e4  caps allow *
> >> 2018-08-20 08:57:58.754 7f9d85934700 20 is_capable service=mon
> >> command= read on cap allow *
> >> 2018-08-20 08:57:58.754 7f9d85934700 20  allow so far , doing grant allow *
> >> 

Re: [ceph-users] Tons of "cls_rgw.cc:3284: gc_iterate_entries end_key=" records in OSD logs

[Yehuda Sadeh-Weinraub]

There was an existing bug reported for this one, and it's fixed on master:

http://tracker.ceph.com/issues/23801

It will be backport to luminous and mimic.

On Mon, Aug 20, 2018 at 9:25 AM, Yehuda Sadeh-Weinraub
 wrote:
> That message has been there since 2014. We should lower the log level though.
>
> Yehuda
>
> On Mon, Aug 20, 2018 at 6:08 AM, David Turner  wrote:
>> In luminous they consolidated a lot of the rgw metadata pools by using
>> namespace inside of the pools. I would say that the GC pool was consolidated
>> into the log pool based on the correlation you've found with the primary
>> osds.  At least that mystery is solved as to why those 8 osds. I don't know
>> why there logs are being spammed with GC messages though. Hopefully someone
>> else can shed a light on that. I cc'd Yehuda on this, the primary RGW dev.
>>
>>
>> On Mon, Aug 20, 2018, 7:09 AM Jakub Jaszewski 
>> wrote:
>>>
>>> Hi David,
>>>
>>> Right, we use this cluster (v12.2.5, fresh installation) for RGW, however,
>>> I don't see default.rgw.gc pool like we have on other cluster which was
>>> upgraded to Luminous, 10.2.9 -> 10.2.10 -> 12.2.2 (I believe that
>>> default.rgw.gc pool is there from the time of setting up RGW on Jewel
>>> version and the pool was automatically created).
>>>
>>> On impacted cluster we have below pools
>>> pool 1 '.rgw.root' replicated size 3 min_size 2 crush_rule 2 object_hash
>>> rjenkins pg_num 8 pgp_num 8 last_change 1499 flags hashpspool stripe_width 0
>>> application rgw
>>> pool 2 'rbd' replicated size 3 min_size 2 crush_rule 2 object_hash
>>> rjenkins pg_num 2048 pgp_num 2048 last_change 2230 flags hashpspool
>>> stripe_width 0 application rbd
>>> pool 3 'default.rgw.control' replicated size 3 min_size 2 crush_rule 2
>>> object_hash rjenkins pg_num 8 pgp_num 8 last_change 1501 flags hashpspool
>>> stripe_width 0 application rgw
>>> pool 4 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 2
>>> object_hash rjenkins pg_num 8 pgp_num 8 last_change 1491 flags hashpspool
>>> stripe_width 0 application rgw
>>> pool 5 'default.rgw.log' replicated size 3 min_size 2 crush_rule 2
>>> object_hash rjenkins pg_num 8 pgp_num 8 last_change 1486 flags hashpspool
>>> stripe_width 0 application rgw
>>> pool 7 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule
>>> 2 object_hash rjenkins pg_num 8 pgp_num 8 last_change 1483 owner
>>> 18446744073709551615 flags hashpspool stripe_width 0 application rgw
>>> pool 8 'default.rgw.buckets.data' erasure size 12 min_size 10 crush_rule 3
>>> object_hash rjenkins pg_num 2048 pgp_num 2048 last_change 2228 flags
>>> hashpspool max_bytes 879609302220800 stripe_width 36864 application rgw
>>> pool 9 'default.rgw.buckets.non-ec' replicated size 3 min_size 2
>>> crush_rule 2 object_hash rjenkins pg_num 8 pgp_num 8 last_change 1506 owner
>>> 18446744073709551615 flags hashpspool stripe_width 0 application rgw
>>>
>>> Eight annoying OSDs match to primary OSDs of PGs that make default.rgw.log
>>> pool.
>>>
>>> Many thanks
>>> Jakub
>>>
>>>
>>> On Mon, Aug 20, 2018 at 11:54 AM David Turner 
>>> wrote:

 I'm assuming you use RGW and that you have a GC pool for RGW. It also
 might beat assumed that your GC pool only has 8 PGs.  Are any of those
 guesses correct?

 On Mon, Aug 20, 2018, 5:13 AM Jakub Jaszewski 
 wrote:
>
> Issue tracker http://tracker.ceph.com/issues/23801.
> Still don't know why only particular OSDs write this information to log
> files.
>
> Jakub
>
> On Wed, Aug 8, 2018 at 12:02 PM Jakub Jaszewski
>  wrote:
>>
>> Hi All, exactly the same story today, same 8 OSDs and a lot of garbage
>> collection objects to process
>>
>> Below is the number of "cls_rgw.cc:3284: gc_iterate_entries end_key="
>> entries per OSD log file
>> hostA:
>>   /var/log/ceph/ceph-osd.58.log
>>   1826467
>> hostB:
>>   /var/log/ceph/ceph-osd.88.log
>>   2924241
>> hostC:
>>   /var/log/ceph/ceph-osd.153.log
>>   581002
>>   /var/log/ceph/ceph-osd.164.log
>>   3278606
>> hostD:
>>   /var/log/ceph/ceph-osd.95.log
>>   1426963
>> hostE:
>>   /var/log/ceph/ceph-osd.4.log
>>   2716914
>>   /var/log/ceph/ceph-osd.53.log
>>   943749
>> hostF:
>>   /var/log/ceph/ceph-osd.172.log
>>   4085334
>>
>>
>> # radosgw-admin gc list --include-all|grep oid |wc -l
>> 302357
>> #
>>
>> Can anyone please explain what is going on ?
>>
>> Thanks!
>> Jakub
>>
>> On Tue, Aug 7, 2018 at 3:03 PM Jakub Jaszewski
>>  wrote:
>>>
>>> Hi,
>>>
>>> 8 out of 192 OSDs in our cluster (version 12.2.5) write plenty of
>>> records like "cls_rgw.cc:3284: gc_iterate_entries end_key=" to the
>>> corresponding log files, e.g.
>>>
>>> 2018-08-07 04:34:06.000585 7fdd8f012700  0 
>>> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries
>>> 

Re: [ceph-users] Tons of "cls_rgw.cc:3284: gc_iterate_entries end_key=" records in OSD logs

[Yehuda Sadeh-Weinraub]

That message has been there since 2014. We should lower the log level though.

Yehuda

On Mon, Aug 20, 2018 at 6:08 AM, David Turner  wrote:
> In luminous they consolidated a lot of the rgw metadata pools by using
> namespace inside of the pools. I would say that the GC pool was consolidated
> into the log pool based on the correlation you've found with the primary
> osds.  At least that mystery is solved as to why those 8 osds. I don't know
> why there logs are being spammed with GC messages though. Hopefully someone
> else can shed a light on that. I cc'd Yehuda on this, the primary RGW dev.
>
>
> On Mon, Aug 20, 2018, 7:09 AM Jakub Jaszewski 
> wrote:
>>
>> Hi David,
>>
>> Right, we use this cluster (v12.2.5, fresh installation) for RGW, however,
>> I don't see default.rgw.gc pool like we have on other cluster which was
>> upgraded to Luminous, 10.2.9 -> 10.2.10 -> 12.2.2 (I believe that
>> default.rgw.gc pool is there from the time of setting up RGW on Jewel
>> version and the pool was automatically created).
>>
>> On impacted cluster we have below pools
>> pool 1 '.rgw.root' replicated size 3 min_size 2 crush_rule 2 object_hash
>> rjenkins pg_num 8 pgp_num 8 last_change 1499 flags hashpspool stripe_width 0
>> application rgw
>> pool 2 'rbd' replicated size 3 min_size 2 crush_rule 2 object_hash
>> rjenkins pg_num 2048 pgp_num 2048 last_change 2230 flags hashpspool
>> stripe_width 0 application rbd
>> pool 3 'default.rgw.control' replicated size 3 min_size 2 crush_rule 2
>> object_hash rjenkins pg_num 8 pgp_num 8 last_change 1501 flags hashpspool
>> stripe_width 0 application rgw
>> pool 4 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 2
>> object_hash rjenkins pg_num 8 pgp_num 8 last_change 1491 flags hashpspool
>> stripe_width 0 application rgw
>> pool 5 'default.rgw.log' replicated size 3 min_size 2 crush_rule 2
>> object_hash rjenkins pg_num 8 pgp_num 8 last_change 1486 flags hashpspool
>> stripe_width 0 application rgw
>> pool 7 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule
>> 2 object_hash rjenkins pg_num 8 pgp_num 8 last_change 1483 owner
>> 18446744073709551615 flags hashpspool stripe_width 0 application rgw
>> pool 8 'default.rgw.buckets.data' erasure size 12 min_size 10 crush_rule 3
>> object_hash rjenkins pg_num 2048 pgp_num 2048 last_change 2228 flags
>> hashpspool max_bytes 879609302220800 stripe_width 36864 application rgw
>> pool 9 'default.rgw.buckets.non-ec' replicated size 3 min_size 2
>> crush_rule 2 object_hash rjenkins pg_num 8 pgp_num 8 last_change 1506 owner
>> 18446744073709551615 flags hashpspool stripe_width 0 application rgw
>>
>> Eight annoying OSDs match to primary OSDs of PGs that make default.rgw.log
>> pool.
>>
>> Many thanks
>> Jakub
>>
>>
>> On Mon, Aug 20, 2018 at 11:54 AM David Turner 
>> wrote:
>>>
>>> I'm assuming you use RGW and that you have a GC pool for RGW. It also
>>> might beat assumed that your GC pool only has 8 PGs.  Are any of those
>>> guesses correct?
>>>
>>> On Mon, Aug 20, 2018, 5:13 AM Jakub Jaszewski 
>>> wrote:

 Issue tracker http://tracker.ceph.com/issues/23801.
 Still don't know why only particular OSDs write this information to log
 files.

 Jakub

 On Wed, Aug 8, 2018 at 12:02 PM Jakub Jaszewski
  wrote:
>
> Hi All, exactly the same story today, same 8 OSDs and a lot of garbage
> collection objects to process
>
> Below is the number of "cls_rgw.cc:3284: gc_iterate_entries end_key="
> entries per OSD log file
> hostA:
>   /var/log/ceph/ceph-osd.58.log
>   1826467
> hostB:
>   /var/log/ceph/ceph-osd.88.log
>   2924241
> hostC:
>   /var/log/ceph/ceph-osd.153.log
>   581002
>   /var/log/ceph/ceph-osd.164.log
>   3278606
> hostD:
>   /var/log/ceph/ceph-osd.95.log
>   1426963
> hostE:
>   /var/log/ceph/ceph-osd.4.log
>   2716914
>   /var/log/ceph/ceph-osd.53.log
>   943749
> hostF:
>   /var/log/ceph/ceph-osd.172.log
>   4085334
>
>
> # radosgw-admin gc list --include-all|grep oid |wc -l
> 302357
> #
>
> Can anyone please explain what is going on ?
>
> Thanks!
> Jakub
>
> On Tue, Aug 7, 2018 at 3:03 PM Jakub Jaszewski
>  wrote:
>>
>> Hi,
>>
>> 8 out of 192 OSDs in our cluster (version 12.2.5) write plenty of
>> records like "cls_rgw.cc:3284: gc_iterate_entries end_key=" to the
>> corresponding log files, e.g.
>>
>> 2018-08-07 04:34:06.000585 7fdd8f012700  0 
>> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries
>> end_key=1_01533616446.000580407
>> 2018-08-07 04:34:06.001888 7fdd8f012700  0 
>> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries
>> end_key=1_01533616446.001886318
>> 2018-08-07 04:34:06.003395 7fdd8f012700  0 
>> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries
>> 

Re: [ceph-users] cephfs client version in RedHat/CentOS 7.5

[Ilya Dryomov]

On Mon, Aug 20, 2018 at 4:52 PM Dietmar Rieder
 wrote:
>
> Hi Cephers,
>
>
> I wonder if the cephfs client in RedHat/CentOS 7.5 will be updated to
> luminous?
> As far as I see there is some luminous related stuff that was
> backported, however,
> the "ceph features" command just reports "jewel" as release of my cephfs
> clients running CentOS 7.5 (kernel 3.10.0-862.11.6.el7.x86_64)
>
>
> {
> "mon": {
> "group": {
> "features": "0x3ffddff8eea4fffb",
> "release": "luminous",
> "num": 3
> }
> },
> "mds": {
> "group": {
> "features": "0x3ffddff8eea4fffb",
> "release": "luminous",
> "num": 3
> }
> },
> "osd": {
> "group": {
> "features": "0x3ffddff8eea4fffb",
> "release": "luminous",
> "num": 240
> }
> },
> "client": {
> "group": {
> "features": "0x7010fb86aa42ada",
> "release": "jewel",
> "num": 23
> },
> "group": {
> "features": "0x3ffddff8eea4fffb",
> "release": "luminous",
> "num": 4
> }
> }
> }
>
>
> This prevents me to run ceph balancer using the upmap mode.
>
>
> Any idea?

Hi Dietmar,

All luminous features are supported in RedHat/CentOS 7.5, but it shows
up as jewel due to a technicality.  Just do

  $ ceph osd set-require-min-compat-client luminous --yes-i-really-mean-it

to override the safety check.

See https://www.spinics.net/lists/ceph-users/msg45071.html for details.
It references an upstream kernel, but both the problem and the solution
are the same.

Thanks,

Ilya
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Still risky to remove RBD-Images?

[Mehmet]

Hello,

AFAIK removing of big RBD-Images would lead ceph to produce blocked 
requests - I dont mean caused by poor disks.


Is this still the case with "Luminous (12.2.4)"?

I have a a few images with

- 2 Terrabyte
- 5 Terrabyte
and
- 20 Terrabyte

in size and have to delete the images.

Would be nice if you could enlightne me :)

- Mehmet
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Removing all rados objects based on a prefix

[David Turner]

The general talk about the rados cleanup command is to clean things up
after benchmarking.  Could this command also be used for deleting an old
RGW bucket or an RBD.  For instance, a bucket with a prefix of
`25ff9eff-058b-41e3-8724-cfffecb979c0.9709451.1` such that all objects in
the default.rgw.buckets.data pool for that bucket start with that string.
Could I run [1] this command to clean all of those up?  Listing the full
pool contents and grepping out for that string returns 100M objects and
every way I've come up with to iterate over that list will take us about a
month to get through it.  I would think this has a decent chance to work,
except for the description of the [2] cleanup option from the rados man
page.

Perhaps I'm also barking up the wrong tree.  Does anyone have a better way
to delete large RBDs or buckets?


[1] rados -p default.rgw.buckets.data cleanup --prefix
25ff9eff-058b-41e3-8724-cfffecb979c0.9709451.1

[2] cleanup [ --run-name run_name ] [ --prefix prefix ]
Clean up a previous benchmark operation.  Note: the default
run-name is "benchmark_last_metadata"
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Mimic osd fails to start.

[Alfredo Deza]

On Mon, Aug 20, 2018 at 10:23 AM, Daznis  wrote:
> Hello,
>
> It appears that something is horribly wrong with the cluster itself. I
> can't create or add any new osds to it at all.

Have you added new monitors? Or replaced monitors? I would check that
all your versions match, something seems to be expecting different
versions.

The "Invalid argument" problem is a common thing we see when that happens.

Something that might help a bit here is if you run ceph-medic against
your cluster:

http://docs.ceph.com/ceph-medic/master/



> On Mon, Aug 20, 2018 at 11:04 AM Daznis  wrote:
>>
>> Hello,
>>
>>
>> Zapping the journal didn't help. I tried to create the journal after
>> zapping it. Also failed. I'm not really sure why this happens.
>>
>> Looking at the monitor logs with 20/20 debug I'm seeing these errors:
>>
>> 2018-08-20 08:57:58.753 7f9d85934700  0 mon.mon02@1(peon) e4
>> handle_command mon_command({"prefix": "osd crush set-device-class",
>> "class": "ssd", "ids": ["48"]} v 0) v1
>> 2018-08-20 08:57:58.753 7f9d85934700 20 is_capable service=osd
>> command=osd crush set-device-class read write on cap allow profile osd
>> 2018-08-20 08:57:58.753 7f9d85934700 20  allow so far , doing grant
>> allow profile osd
>> 2018-08-20 08:57:58.753 7f9d85934700 20  match
>> 2018-08-20 08:57:58.753 7f9d85934700 10 mon.mon02@1(peon) e4
>> _allowed_command capable
>> 2018-08-20 08:57:58.753 7f9d85934700  0 log_channel(audit) log [INF] :
>> from='osd.48 10.24.52.17:6800/153683' entity='osd.48' cmd=[{"prefix":
>> "osd crush set-device-class", "class": "ssd", "ids": ["48"]}]:
>> dispatch
>> 2018-08-20 08:57:58.753 7f9d85934700 10 mon.mon02@1(peon).osd e46327
>> preprocess_query mon_command({"prefix": "osd crush set-device-class",
>> "class": "ssd", "ids": ["48"]} v 0) v1 from osd.48
>> 10.24.52.17:6800/153683
>> 2018-08-20 08:57:58.753 7f9d85934700 10 mon.mon02@1(peon) e4
>> forward_request 4 request mon_command({"prefix": "osd crush
>> set-device-class", "class": "ssd", "ids": ["48"]} v 0) v1 features
>> 4611087854031142907
>> 2018-08-20 08:57:58.753 7f9d85934700 20 mon.mon02@1(peon) e4
>> _ms_dispatch existing session 0x55b4ec482a80 for mon.1
>> 10.24.52.11:6789/0
>> 2018-08-20 08:57:58.753 7f9d85934700 20 mon.mon02@1(peon) e4  caps allow *
>> 2018-08-20 08:57:58.753 7f9d85934700 10 mon.mon02@1(peon).log
>> v10758065 preprocess_query log(1 entries from seq 4 at 2018-08-20
>> 08:57:58.755306) v1 from mon.1 10.24.52.11:6789/0
>> 2018-08-20 08:57:58.753 7f9d85934700 10 mon.mon02@1(peon).log
>> v10758065 preprocess_log log(1 entries from seq 4 at 2018-08-20
>> 08:57:58.755306) v1 from mon.1
>> 2018-08-20 08:57:58.753 7f9d85934700 20 is_capable service=log
>> command= write on cap allow *
>> 2018-08-20 08:57:58.753 7f9d85934700 20  allow so far , doing grant allow *
>> 2018-08-20 08:57:58.753 7f9d85934700 20  allow all
>> 2018-08-20 08:57:58.753 7f9d85934700 10 mon.mon02@1(peon) e4
>> forward_request 5 request log(1 entries from seq 4 at 2018-08-20
>> 08:57:58.755306) v1 features 4611087854031142907
>> 2018-08-20 08:57:58.754 7f9d85934700 20 mon.mon02@1(peon) e4
>> _ms_dispatch existing session 0x55b4ec4828c0 for mon.0
>> 10.24.52.10:6789/0
>> 2018-08-20 08:57:58.754 7f9d85934700 20 mon.mon02@1(peon) e4  caps allow *
>> 2018-08-20 08:57:58.754 7f9d85934700 20 is_capable service=mon
>> command= read on cap allow *
>> 2018-08-20 08:57:58.754 7f9d85934700 20  allow so far , doing grant allow *
>> 2018-08-20 08:57:58.754 7f9d85934700 20  allow all
>> 2018-08-20 08:57:58.754 7f9d85934700 20 is_capable service=mon
>> command= exec on cap allow *
>> 2018-08-20 08:57:58.754 7f9d85934700 20  allow so far , doing grant allow *
>> 2018-08-20 08:57:58.754 7f9d85934700 20  allow all
>> 2018-08-20 08:57:58.754 7f9d85934700 10 mon.mon02@1(peon) e4
>> handle_route mon_command_ack([{"prefix": "osd crush set-device-class",
>> "class": "ssd", "ids": ["48"]}]=-22 (22) Invalid argument v46327) v1
>> to unknown.0 -
>> 2018-08-20 08:57:58.785 7f9d85934700 10 mon.mon02@1(peon) e4
>> ms_handle_reset 0x55b4ecf4b200 10.24.52.17:6800/153683
>> 2018-08-20 08:57:58.785 7f9d85934700 10 mon.mon02@1(peon) e4
>> reset/close on session osd.48 10.24.52.17:6800/153683
>> 2018-08-20 08:57:58.785 7f9d85934700 10 mon.mon02@1(peon) e4
>> remove_session 0x55b4ecf86380 osd.48 10.24.52.17:6800/153683 features
>> 0x3ffddff8ffa4fffb
>> 2018-08-20 08:57:58.828 7f9d85934700 20 mon.mon02@1(peon) e4
>> _ms_dispatch existing session 0x55b4ec4828c0 for mon.0
>> 10.24.52.10:6789/0
>> On Sat, Aug 18, 2018 at 7:54 PM Daznis  wrote:
>> >
>> > Hello,
>> >
>> > not sure about it. I assumed ceph-deploy would do it with the
>> > "--zap-disk" flag defined. I will try it on Monday and report the
>> > progress.
>> > On Sat, Aug 18, 2018 at 3:02 PM Alfredo Deza  wrote:
>> > >
>> > > On Fri, Aug 17, 2018 at 7:05 PM, Daznis  wrote:
>> > > > Hello,
>> > > >
>> > > >
>> > > > I have replace one of our failed OSD drives and recreated a new osd
>> > > > with ceph-deploy and it failes to start.

[ceph-users] cephfs client version in RedHat/CentOS 7.5

[Dietmar Rieder]

Hi Cephers,


I wonder if the cephfs client in RedHat/CentOS 7.5 will be updated to
luminous?
As far as I see there is some luminous related stuff that was
backported, however,
the "ceph features" command just reports "jewel" as release of my cephfs
clients running CentOS 7.5 (kernel 3.10.0-862.11.6.el7.x86_64)


{
"mon": {
"group": {
"features": "0x3ffddff8eea4fffb",
"release": "luminous",
"num": 3
}
},
"mds": {
"group": {
"features": "0x3ffddff8eea4fffb",
"release": "luminous",
"num": 3
}
},
"osd": {
"group": {
"features": "0x3ffddff8eea4fffb",
"release": "luminous",
"num": 240
}
},
"client": {
"group": {
"features": "0x7010fb86aa42ada",
"release": "jewel",
"num": 23
},
"group": {
"features": "0x3ffddff8eea4fffb",
"release": "luminous",
"num": 4
}
}
}


This prevents me to run ceph balancer using the upmap mode.


Any idea?

  Dietmar



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Set existing pools to use hdd device class only

[Eugen Block]

The correct URL should be:

http://lists.ceph.com/pipermail/ceph-large-ceph.com/2018-April/000106.html


Zitat von Jonathan Proulx :


On Mon, Aug 20, 2018 at 06:13:26AM -0400, David Turner wrote:

:There is a thread from the ceph-large users ML that covered a way to do
:this change without shifting data for an HDD only cluster.  Hopefully it
:will be helpful for you.
:
:
:httpists.ceph.com/pipermail/ceph-large-ceph.com/2018-April/000106.html

Wish I knew that list existed before I thrashed my entire cloud for a
week doing exactly this. 95% data move on a cluster that already
experiencing IO exhaustion was not fun. Luckily it was "only" 0.5PB

-Jon

:On Mon, Aug 20, 2018, 5:42 AM Enrico Kern  wrote:
:
:> Hmm then is not really an option for me. Maybe someone from the devs can
:> shed a light why it is doing migration as long you only have OSDs with the
:> same class? I have a few Petabyte of Storage in each cluster. When it
:> starts migrating everything again that will result in a super big
:> performance bottleneck. My plan was to set the existing pools to use the
:> new crush rule hdd only and add ssd osds later.
:>
:> On Mon, Aug 20, 2018 at 11:22 AM Marc Roos 
:> wrote:
:>
:>>
:>> I just recently did the same. Take into account that everything starts
:>> migrating. How weird it maybe, I had hdd test cluster only and changed
:>> the crush rule to having hdd. Took a few days, totally unnecessary as
:>> far as I am concerned.
:>>
:>>
:>>
:>>
:>> -Original Message-
:>> From: Enrico Kern [mailto:enrico.k...@glispa.com]
:>> Sent: maandag 20 augustus 2018 11:18
:>> To: ceph-users@lists.ceph.com
:>> Subject: [ceph-users] Set existing pools to use hdd device class only
:>>
:>> Hello,
:>>
:>> right now we have multiple HDD only clusters with ether filestore
:>> journals on SSDs or on newer installations WAL etc. on SSD.
:>>
:>> I plan to extend our ceph clusters with SSDs to provide ssd only pools.
:>> In luminous we have devices classes so that i should be able todo this
:>> without editing around crush map.
:>>
:>> In the device class doc it says i can create "new" pools to use only
:>> ssds as example:
:>>
:>> ceph osd crush rule create-replicated fast default host ssd
:>>
:>> what happens if i fire this on an existing pool but with hdd device
:>> class? I wasnt able to test thi yet in our staging cluster and wanted to
:>> ask whats the way todo this.
:>>
:>> I want to set an existing pool called volumes to only use osds with hdd
:>> class. Right now all OSDs have hdds. So in theory it should not use
:>> newly created SSD osds once i set them all up for hdd classes right?
:>>
:>> So for existing pool running:
:>>
:>> ceph osd crush rule create-replicated volume-hdd-only volumes default
:>> hdd ceph osd pool set volumes crush_rule volume-hdd-only
:>>
:>>
:>> should be the way to go right?
:>>
:>>
:>> Regards,
:>>
:>> Enrico
:>>
:>> --
:>>
:>>
:>> Enrico Kern
:>>
:>> VP IT Operations
:>>
:>>
:>> enrico.k...@glispa.com
:>> +49 (0) 30 555713017 / +49 (0) 152 26814501
:>>
:>> skype: flyersa
:>> LinkedIn Profile 
:>>
:>>
:>>   
:>>
:>>
:>> Glispa GmbH | Berlin Office
:>>
:>> Stromstr. 11-17  
:>> Berlin, Germany, 10551  
:>>
:>> Managing Director Din Karol-Gavish
:>> Registered in Berlin
:>> AG Charlottenburg |
:>> <
:>>  
https://maps.google.com/?q=Sonnenburger+Stra%C3%9Fe+73+10437+Berlin%C2%A0%7C+%3Chttps://maps.google.com/?q%3DSonnenburgerstra%25C3%259Fe%2B73%2B10437%2BBerlin%25C2%25A0%257C%25C2%25A0Germany%26entry%3Dgmail%26source%3Dg%3E%C2%A0Germany=gmail=g>

:>>
:>> HRB 114678B –
:>>
:>>
:>>
:>
:> --
:>
:> *Enrico Kern*
:> VP IT Operations
:>
:> enrico.k...@glispa.com
:> +49 (0) 30 555713017 / +49 (0) 152 26814501
:> skype: flyersa
:> LinkedIn Profile 
:>
:>
:>  
:>
:> *Glispa GmbH* | Berlin Office
:> Stromstr. 11-17  
:> Berlin, Germany, 10551  
:>
:> Managing Director Din Karol-Gavish
:> Registered in Berlin
:> AG Charlottenburg |
:>  
  
HRB

:> 114678B
:> –
:> ___
:> ceph-users mailing list
:> ceph-users@lists.ceph.com
:> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
:>

:___
:ceph-users mailing list
:ceph-users@lists.ceph.com
:http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


--
___
ceph-users mailing list
ceph-users@lists.ceph.com

Re: [ceph-users] Set existing pools to use hdd device class only

[Jonathan Proulx]

On Mon, Aug 20, 2018 at 06:13:26AM -0400, David Turner wrote:

:There is a thread from the ceph-large users ML that covered a way to do
:this change without shifting data for an HDD only cluster.  Hopefully it
:will be helpful for you.
:
:
:httpists.ceph.com/pipermail/ceph-large-ceph.com/2018-April/000106.html

Wish I knew that list existed before I thrashed my entire cloud for a
week doing exactly this. 95% data move on a cluster that already
experiencing IO exhaustion was not fun. Luckily it was "only" 0.5PB

-Jon

:On Mon, Aug 20, 2018, 5:42 AM Enrico Kern  wrote:
:
:> Hmm then is not really an option for me. Maybe someone from the devs can
:> shed a light why it is doing migration as long you only have OSDs with the
:> same class? I have a few Petabyte of Storage in each cluster. When it
:> starts migrating everything again that will result in a super big
:> performance bottleneck. My plan was to set the existing pools to use the
:> new crush rule hdd only and add ssd osds later.
:>
:> On Mon, Aug 20, 2018 at 11:22 AM Marc Roos 
:> wrote:
:>
:>>
:>> I just recently did the same. Take into account that everything starts
:>> migrating. How weird it maybe, I had hdd test cluster only and changed
:>> the crush rule to having hdd. Took a few days, totally unnecessary as
:>> far as I am concerned.
:>>
:>>
:>>
:>>
:>> -Original Message-
:>> From: Enrico Kern [mailto:enrico.k...@glispa.com]
:>> Sent: maandag 20 augustus 2018 11:18
:>> To: ceph-users@lists.ceph.com
:>> Subject: [ceph-users] Set existing pools to use hdd device class only
:>>
:>> Hello,
:>>
:>> right now we have multiple HDD only clusters with ether filestore
:>> journals on SSDs or on newer installations WAL etc. on SSD.
:>>
:>> I plan to extend our ceph clusters with SSDs to provide ssd only pools.
:>> In luminous we have devices classes so that i should be able todo this
:>> without editing around crush map.
:>>
:>> In the device class doc it says i can create "new" pools to use only
:>> ssds as example:
:>>
:>> ceph osd crush rule create-replicated fast default host ssd
:>>
:>> what happens if i fire this on an existing pool but with hdd device
:>> class? I wasnt able to test thi yet in our staging cluster and wanted to
:>> ask whats the way todo this.
:>>
:>> I want to set an existing pool called volumes to only use osds with hdd
:>> class. Right now all OSDs have hdds. So in theory it should not use
:>> newly created SSD osds once i set them all up for hdd classes right?
:>>
:>> So for existing pool running:
:>>
:>> ceph osd crush rule create-replicated volume-hdd-only volumes default
:>> hdd ceph osd pool set volumes crush_rule volume-hdd-only
:>>
:>>
:>> should be the way to go right?
:>>
:>>
:>> Regards,
:>>
:>> Enrico
:>>
:>> --
:>>
:>>
:>> Enrico Kern
:>>
:>> VP IT Operations
:>>
:>>
:>> enrico.k...@glispa.com
:>> +49 (0) 30 555713017 / +49 (0) 152 26814501
:>>
:>> skype: flyersa
:>> LinkedIn Profile 
:>>
:>>
:>>   
:>>
:>>
:>> Glispa GmbH | Berlin Office
:>>
:>> Stromstr. 11-17  
:>> Berlin, Germany, 10551  
:>>
:>> Managing Director Din Karol-Gavish
:>> Registered in Berlin
:>> AG Charlottenburg |
:>> <
:>> 
https://maps.google.com/?q=Sonnenburger+Stra%C3%9Fe+73+10437+Berlin%C2%A0%7C+%3Chttps://maps.google.com/?q%3DSonnenburgerstra%25C3%259Fe%2B73%2B10437%2BBerlin%25C2%25A0%257C%25C2%25A0Germany%26entry%3Dgmail%26source%3Dg%3E%C2%A0Germany=gmail=g>
:>>
:>> HRB 114678B –
:>>
:>>
:>>
:>
:> --
:>
:> *Enrico Kern*
:> VP IT Operations
:>
:> enrico.k...@glispa.com
:> +49 (0) 30 555713017 / +49 (0) 152 26814501
:> skype: flyersa
:> LinkedIn Profile 
:>
:>
:>  
:>
:> *Glispa GmbH* | Berlin Office
:> Stromstr. 11-17  
:> Berlin, Germany, 10551  
:>
:> Managing Director Din Karol-Gavish
:> Registered in Berlin
:> AG Charlottenburg |
:> 

 HRB
:> 114678B
:> –
:> ___
:> ceph-users mailing list
:> ceph-users@lists.ceph.com
:> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
:>

:___
:ceph-users mailing list
:ceph-users@lists.ceph.com
:http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


-- 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ensure Hammer client compatibility

[Lincoln Bryant]

Hi Kees,

What interfaces do your Hammer clients need? If you're looking at
CephFS, we have had reasonable success moving our older clients (EL6)
to NFS Ganesha with the Ceph FSAL.

--Lincoln

On Mon, 2018-08-20 at 12:22 +0200, Kees Meijs wrote:
> Good afternoon Cephers,
> 
> While I'm fixing our upgrade-semi-broken cluster (see thread Upgrade
> to Infernalis: failed to pick suitable auth object) I'm wondering
> about ensuring client compatibility.
> 
> My end goal is BlueStore (i.e. running Luminous) and unfortunately
> I'm obliged to offer Hammer client compatibility.
> 
> Any pointers on how to ensure this configuration-wise?
> 
> Thanks!
> 
> Regards,
> Kees
> 
> -- 
> https://nefos.nl/contact 
> 
> Nefos IT bv
> Ambachtsweg 25 (industrienummer 4217)
> 5627 BZ Eindhoven
> Nederland
> 
> KvK 66494931
> 
> Aanwezig op maandag, dinsdag, woensdag en vrijdag
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Mimic osd fails to start.

[Daznis]

Hello,

It appears that something is horribly wrong with the cluster itself. I
can't create or add any new osds to it at all.
On Mon, Aug 20, 2018 at 11:04 AM Daznis  wrote:
>
> Hello,
>
>
> Zapping the journal didn't help. I tried to create the journal after
> zapping it. Also failed. I'm not really sure why this happens.
>
> Looking at the monitor logs with 20/20 debug I'm seeing these errors:
>
> 2018-08-20 08:57:58.753 7f9d85934700  0 mon.mon02@1(peon) e4
> handle_command mon_command({"prefix": "osd crush set-device-class",
> "class": "ssd", "ids": ["48"]} v 0) v1
> 2018-08-20 08:57:58.753 7f9d85934700 20 is_capable service=osd
> command=osd crush set-device-class read write on cap allow profile osd
> 2018-08-20 08:57:58.753 7f9d85934700 20  allow so far , doing grant
> allow profile osd
> 2018-08-20 08:57:58.753 7f9d85934700 20  match
> 2018-08-20 08:57:58.753 7f9d85934700 10 mon.mon02@1(peon) e4
> _allowed_command capable
> 2018-08-20 08:57:58.753 7f9d85934700  0 log_channel(audit) log [INF] :
> from='osd.48 10.24.52.17:6800/153683' entity='osd.48' cmd=[{"prefix":
> "osd crush set-device-class", "class": "ssd", "ids": ["48"]}]:
> dispatch
> 2018-08-20 08:57:58.753 7f9d85934700 10 mon.mon02@1(peon).osd e46327
> preprocess_query mon_command({"prefix": "osd crush set-device-class",
> "class": "ssd", "ids": ["48"]} v 0) v1 from osd.48
> 10.24.52.17:6800/153683
> 2018-08-20 08:57:58.753 7f9d85934700 10 mon.mon02@1(peon) e4
> forward_request 4 request mon_command({"prefix": "osd crush
> set-device-class", "class": "ssd", "ids": ["48"]} v 0) v1 features
> 4611087854031142907
> 2018-08-20 08:57:58.753 7f9d85934700 20 mon.mon02@1(peon) e4
> _ms_dispatch existing session 0x55b4ec482a80 for mon.1
> 10.24.52.11:6789/0
> 2018-08-20 08:57:58.753 7f9d85934700 20 mon.mon02@1(peon) e4  caps allow *
> 2018-08-20 08:57:58.753 7f9d85934700 10 mon.mon02@1(peon).log
> v10758065 preprocess_query log(1 entries from seq 4 at 2018-08-20
> 08:57:58.755306) v1 from mon.1 10.24.52.11:6789/0
> 2018-08-20 08:57:58.753 7f9d85934700 10 mon.mon02@1(peon).log
> v10758065 preprocess_log log(1 entries from seq 4 at 2018-08-20
> 08:57:58.755306) v1 from mon.1
> 2018-08-20 08:57:58.753 7f9d85934700 20 is_capable service=log
> command= write on cap allow *
> 2018-08-20 08:57:58.753 7f9d85934700 20  allow so far , doing grant allow *
> 2018-08-20 08:57:58.753 7f9d85934700 20  allow all
> 2018-08-20 08:57:58.753 7f9d85934700 10 mon.mon02@1(peon) e4
> forward_request 5 request log(1 entries from seq 4 at 2018-08-20
> 08:57:58.755306) v1 features 4611087854031142907
> 2018-08-20 08:57:58.754 7f9d85934700 20 mon.mon02@1(peon) e4
> _ms_dispatch existing session 0x55b4ec4828c0 for mon.0
> 10.24.52.10:6789/0
> 2018-08-20 08:57:58.754 7f9d85934700 20 mon.mon02@1(peon) e4  caps allow *
> 2018-08-20 08:57:58.754 7f9d85934700 20 is_capable service=mon
> command= read on cap allow *
> 2018-08-20 08:57:58.754 7f9d85934700 20  allow so far , doing grant allow *
> 2018-08-20 08:57:58.754 7f9d85934700 20  allow all
> 2018-08-20 08:57:58.754 7f9d85934700 20 is_capable service=mon
> command= exec on cap allow *
> 2018-08-20 08:57:58.754 7f9d85934700 20  allow so far , doing grant allow *
> 2018-08-20 08:57:58.754 7f9d85934700 20  allow all
> 2018-08-20 08:57:58.754 7f9d85934700 10 mon.mon02@1(peon) e4
> handle_route mon_command_ack([{"prefix": "osd crush set-device-class",
> "class": "ssd", "ids": ["48"]}]=-22 (22) Invalid argument v46327) v1
> to unknown.0 -
> 2018-08-20 08:57:58.785 7f9d85934700 10 mon.mon02@1(peon) e4
> ms_handle_reset 0x55b4ecf4b200 10.24.52.17:6800/153683
> 2018-08-20 08:57:58.785 7f9d85934700 10 mon.mon02@1(peon) e4
> reset/close on session osd.48 10.24.52.17:6800/153683
> 2018-08-20 08:57:58.785 7f9d85934700 10 mon.mon02@1(peon) e4
> remove_session 0x55b4ecf86380 osd.48 10.24.52.17:6800/153683 features
> 0x3ffddff8ffa4fffb
> 2018-08-20 08:57:58.828 7f9d85934700 20 mon.mon02@1(peon) e4
> _ms_dispatch existing session 0x55b4ec4828c0 for mon.0
> 10.24.52.10:6789/0
> On Sat, Aug 18, 2018 at 7:54 PM Daznis  wrote:
> >
> > Hello,
> >
> > not sure about it. I assumed ceph-deploy would do it with the
> > "--zap-disk" flag defined. I will try it on Monday and report the
> > progress.
> > On Sat, Aug 18, 2018 at 3:02 PM Alfredo Deza  wrote:
> > >
> > > On Fri, Aug 17, 2018 at 7:05 PM, Daznis  wrote:
> > > > Hello,
> > > >
> > > >
> > > > I have replace one of our failed OSD drives and recreated a new osd
> > > > with ceph-deploy and it failes to start.
> > >
> > > Is it possible you haven't zapped the journal on nvme0n1p13 ?
> > >
> > >
> > >
> > > >
> > > > Command: ceph-deploy --overwrite-conf osd create --filestore
> > > > --zap-disk --data /dev/bcache0 --journal /dev/nvme0n1p13 
> > > >
> > > > Output off ceph-deploy:
> > > > [ceph_deploy.conf][DEBUG ] found configuration file at: 
> > > > /root/.cephdeploy.conf
> > > > [ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy
> > > > --overwrite-conf osd create --filestore 

Re: [ceph-users] failing to respond to cache pressure

[Eugen Block]

Update: we are getting these messages again.

So the search continues...


Zitat von Eugen Block :


Hi,

Depending on your kernel (memory leaks with CephFS) increasing the  
mds_cache_memory_limit could be of help. What is your current  
setting now?


ceph:~ # ceph daemon mds. config show | grep mds_cache_memory_limit

We had these messages for months, almost every day.
It would occur when hourly backup jobs ran and the MDS had to serve  
an additional client (searching the whole CephFS for changes)  
besides the existing CephFS clients. First we updated all clients to  
a more recent kernel version, but the warnings didn't stop. Then we  
doubled the cache size from 2 GB to 4 GB last week and since then I  
haven't seen this warning again (for now).


Try playing with the cache size to find a setting fitting your  
needs, but don't forget to monitor your MDS in case something goes  
wrong.


Regards,
Eugen


Zitat von Wido den Hollander :


On 08/13/2018 01:22 PM, Zhenshi Zhou wrote:

Hi,
Recently, the cluster runs healthy, but I get warning messages everyday:



Which version of Ceph? Which version of clients?

Can you post:

$ ceph versions
$ ceph features
$ ceph fs status

Wido


2018-08-13 17:39:23.682213 [INF]  Cluster is now healthy
2018-08-13 17:39:23.682144 [INF]  Health check cleared:
MDS_CLIENT_RECALL (was: 6 clients failing to respond to cache pressure)
2018-08-13 17:39:23.052022 [INF]  MDS health message cleared (mds.0):
Client docker38:docker failing to respond to cache pressure
2018-08-13 17:39:23.051979 [INF]  MDS health message cleared (mds.0):
Client docker73:docker failing to respond to cache pressure
2018-08-13 17:39:23.051934 [INF]  MDS health message cleared (mds.0):
Client docker74:docker failing to respond to cache pressure
2018-08-13 17:39:23.051853 [INF]  MDS health message cleared (mds.0):
Client docker75:docker failing to respond to cache pressure
2018-08-13 17:39:23.051815 [INF]  MDS health message cleared (mds.0):
Client docker27:docker failing to respond to cache pressure
2018-08-13 17:39:23.051753 [INF]  MDS health message cleared (mds.0):
Client docker27 failing to respond to cache pressure
2018-08-13 17:38:11.100331 [WRN]  Health check update: 6 clients failing
to respond to cache pressure (MDS_CLIENT_RECALL)
2018-08-13 17:37:39.570014 [WRN]  Health check update: 5 clients failing
to respond to cache pressure (MDS_CLIENT_RECALL)
2018-08-13 17:37:31.099418 [WRN]  Health check update: 3 clients failing
to respond to cache pressure (MDS_CLIENT_RECALL)
2018-08-13 17:36:34.564345 [WRN]  Health check update: 1 clients failing
to respond to cache pressure (MDS_CLIENT_RECALL)
2018-08-13 17:36:27.121891 [WRN]  Health check update: 3 clients failing
to respond to cache pressure (MDS_CLIENT_RECALL)
2018-08-13 17:36:11.967531 [WRN]  Health check update: 5 clients failing
to respond to cache pressure (MDS_CLIENT_RECALL)
2018-08-13 17:35:59.870055 [WRN]  Health check update: 6 clients failing
to respond to cache pressure (MDS_CLIENT_RECALL)
2018-08-13 17:35:47.787323 [WRN]  Health check update: 3 clients failing
to respond to cache pressure (MDS_CLIENT_RECALL)
2018-08-13 17:34:59.435933 [WRN]  Health check failed: 1 clients failing
to respond to cache pressure (MDS_CLIENT_RECALL)
2018-08-13 17:34:59.045510 [WRN]  MDS health message (mds.0): Client
docker75:docker failing to respond to cache pressure  

How can I fix it?


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Questions on CRUSH map

[Cody]

Hi Konstantin,

Thank you for looking into my question.

I was trying to understand how to set up CRUSH hierarchies and set
rules for different failure domains. I am particularly confused by the
'step take' and 'step choose|chooseleaf' settings for which I think
are the keys for defining a failure domain in a CRUSH rule.

As for my hypothetical cluster, it is made of 3 racks with 2 hosts on
each. One host has 3 SSD-based OSDs and the other has 3 HDD-based
OSDs. I wished to create two rules: one uses SSD-only and another
HDD-only. Both rules should have a rack level failure domain.

I have attached a diagram that may help to explain my setup. The
following is my CRUSH map configuration (with all typos fixed) for
review:

device 0 osd.0 class ssd
device 1 osd.1 class ssd
device 2 osd.2 class ssd
device 3 osd.3 class hdd
device 4 osd.4 class hdd
device 5 osd.5 class hdd
device 6 osd.6 class ssd
device 7 osd.7 class ssd
device 8 osd.8 class ssd
device 9 osd.9 class hdd
device 10 osd.10 class hdd
device 11 osd.11 class hdd
device 12 osd.12 class ssd
device 13 osd.13 class ssd
device 14 osd.14 class ssd
device 15 osd.15 class hdd
device 16 osd.17 class hdd
device 17 osd.17 class hdd

  host a1-1 {
  id -1
  alg straw
  hash 0
  item osd.0 weight 1.00
  item osd.1 weight 1.00
  item osd.2 weight 1.00
  }

  host a1-2 {
  id -2
  alg straw
  hash 0
  item osd.3 weight 1.00
  item osd.4 weight 1.00
  item osd.5 weight 1.00
  }

  host a2-1 {
  id -3
  alg straw
  hash 0
  item osd.6 weight 1.00
  item osd.7 weight 1.00
  item osd.8 weight 1.00
  }

  host a2-2 {
  id -4
  alg straw
  hash 0
  item osd.9 weight 1.00
  item osd.10 weight 1.00
  item osd.11 weight 1.00
  }

  host a3-1 {
  id -5
  alg straw
  hash 0
  item osd.12 weight 1.00
  item osd.13 weight 1.00
  item osd.14 weight 1.00
  }

  host a3-2 {
  id -6
  alg straw
  hash 0
  item osd.15 weight 1.00
  item osd.16 weight 1.00
  item osd.17 weight 1.00
  }

  rack a1 {
  id -7
  alg straw
  hash 0
  item a1-1 weight 3.0
  item a1-2 weight 3.0
  }

  rack a2 {
  id -5
  alg straw
  hash 0
  item a2-1 weight 3.0
  item a2-2 weight 3.0
  }

  rack a3 {
  id -6
  alg straw
  hash 0
  item a3-1 weight 3.0
  item a3-2 weight 3.0
  }

  row a {
  id -7
  alg straw
  hash 0
  item a1 6.0
  item a2 6.0
  item a3 6.0
  }

  rule ssd {
  id 1
  type replicated
  min_size 2
  max_size 11
  step take a class ssd
  step chooseleaf firstn 0 type rack
  step emit
  }

  rule hdd {
  id 2
  type replicated
  min_size 2
  max_size 11
  step take a class hdd
  step chooseleaf firstn 0 type rack
  step emit
  }


Are the two rules correct?

Regards,
Cody
On Sun, Aug 19, 2018 at 11:55 PM Konstantin Shalygin  wrote:
>
> > Hi everyone,
> >
> > I am new to Ceph and trying to test out my understanding on the CRUSH
> > map. Attached is a hypothetical cluster diagram with 3 racks. On each
> > rack, the first host runs 3 SSD-based OSDs and the second 3 HDD-based.
> >
> > My goal is to create two rules that separate SSD and HDD performance
> > domains (by using device class) and both rules should use a*rack*
> > level failure domain.
>
>
> Please, provide on human language what actually hosts configuration you
> have and what drive pattern usage you want at finish?
>
> Obviously you want just separate your ssd/hdd load per pool basis, but I
> need more information to clarify.
>
>
>
>
> k
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Tons of "cls_rgw.cc:3284: gc_iterate_entries end_key=" records in OSD logs

[David Turner]

In luminous they consolidated a lot of the rgw metadata pools by using
namespace inside of the pools. I would say that the GC pool was
consolidated into the log pool based on the correlation you've found with
the primary osds.  At least that mystery is solved as to why those 8 osds.
I don't know why there logs are being spammed with GC messages though.
Hopefully someone else can shed a light on that. I cc'd Yehuda on this, the
primary RGW dev.

On Mon, Aug 20, 2018, 7:09 AM Jakub Jaszewski 
wrote:

> Hi David,
>
> Right, we use this cluster (v12.2.5, fresh installation) for RGW, however,
> I don't see default.rgw.gc pool like we have on other cluster which was
> upgraded to Luminous, 10.2.9 -> 10.2.10 -> 12.2.2 (I believe that
> default.rgw.gc pool is there from the time of setting up RGW on Jewel
> version and the pool was automatically created).
>
> On impacted cluster we have below pools
> pool 1 '.rgw.root' replicated size 3 min_size 2 crush_rule 2 object_hash
> rjenkins pg_num 8 pgp_num 8 last_change 1499 flags hashpspool stripe_width
> 0 application rgw
> pool 2 'rbd' replicated size 3 min_size 2 crush_rule 2 object_hash
> rjenkins pg_num 2048 pgp_num 2048 last_change 2230 flags hashpspool
> stripe_width 0 application rbd
> pool 3 'default.rgw.control' replicated size 3 min_size 2 crush_rule 2
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 1501 flags hashpspool
> stripe_width 0 application rgw
> pool 4 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 2
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 1491 flags hashpspool
> stripe_width 0 application rgw
> pool 5 'default.rgw.log' replicated size 3 min_size 2 crush_rule 2
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 1486 flags hashpspool
> stripe_width 0 application rgw
> pool 7 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule
> 2 object_hash rjenkins pg_num 8 pgp_num 8 last_change 1483 owner
> 18446744073709551615 flags hashpspool stripe_width 0 application rgw
> pool 8 'default.rgw.buckets.data' erasure size 12 min_size 10 crush_rule 3
> object_hash rjenkins pg_num 2048 pgp_num 2048 last_change 2228 flags
> hashpspool max_bytes 879609302220800 stripe_width 36864 application rgw
> pool 9 'default.rgw.buckets.non-ec' replicated size 3 min_size 2
> crush_rule 2 object_hash rjenkins pg_num 8 pgp_num 8 last_change 1506 owner
> 18446744073709551615 flags hashpspool stripe_width 0 application rgw
>
> Eight annoying OSDs match to primary OSDs of PGs that make default.rgw.log
> pool.
>
> Many thanks
> Jakub
>
>
> On Mon, Aug 20, 2018 at 11:54 AM David Turner 
> wrote:
>
>> I'm assuming you use RGW and that you have a GC pool for RGW. It also
>> might beat assumed that your GC pool only has 8 PGs.  Are any of those
>> guesses correct?
>>
>> On Mon, Aug 20, 2018, 5:13 AM Jakub Jaszewski 
>> wrote:
>>
>>> Issue tracker http://tracker.ceph.com/issues/23801.
>>> Still don't know why only particular OSDs write this information to log
>>> files.
>>>
>>> Jakub
>>>
>>> On Wed, Aug 8, 2018 at 12:02 PM Jakub Jaszewski <
>>> jaszewski.ja...@gmail.com> wrote:
>>>
 Hi All, exactly the same story today, same 8 OSDs and a lot of garbage
 collection objects to process

 Below is the number of "cls_rgw.cc:3284: gc_iterate_entries end_key="
 entries per OSD log file
 hostA:
   /var/log/ceph/ceph-osd.58.log
   1826467
 hostB:
   /var/log/ceph/ceph-osd.88.log
   2924241
 hostC:
   /var/log/ceph/ceph-osd.153.log
   581002
   /var/log/ceph/ceph-osd.164.log
   3278606
 hostD:
   /var/log/ceph/ceph-osd.95.log
   1426963
 hostE:
   /var/log/ceph/ceph-osd.4.log
   2716914
   /var/log/ceph/ceph-osd.53.log
   943749
 hostF:
   /var/log/ceph/ceph-osd.172.log
   4085334


 # radosgw-admin gc list --include-all|grep oid |wc -l
 302357
 #

 Can anyone please explain what is going on ?

 Thanks!
 Jakub

 On Tue, Aug 7, 2018 at 3:03 PM Jakub Jaszewski <
 jaszewski.ja...@gmail.com> wrote:

> Hi,
>
> 8 out of 192 OSDs in our cluster (version 12.2.5) write plenty of
> records like "cls_rgw.cc:3284: gc_iterate_entries end_key=" to the
> corresponding log files, e.g.
>
> 2018-08-07 04:34:06.000585 7fdd8f012700  0 
> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries
> end_key=1_01533616446.000580407
> 2018-08-07 04:34:06.001888 7fdd8f012700  0 
> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries
> end_key=1_01533616446.001886318
> 2018-08-07 04:34:06.003395 7fdd8f012700  0 
> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries
> end_key=1_01533616446.003390299
> 2018-08-07 04:34:06.005205 7fdd8f012700  0 
> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries
> end_key=1_01533616446.005200341
>
> # grep '2018-08-07 04:34:06' 

Re: [ceph-users] FreeBSD rc.d script: sta.rt not found

[Norman Gray]

Willem Jan, hello.

On 16 Aug 2018, at 12:07, Willem Jan Withagen wrote:

In the mean time I have uploaded a PR to fix this in the manual, which 
should read:

  gpart create -s GPT ada1
  gpart add -t freebsd-zfs -l osd.1 ada1
  zpool create osd.1 gpt/osd.1
  zfs create -o mountpoint=/var/lib/ceph/osd/osd.1 osd.1


I'm still having some difficulties with these instructions, I'm afraid.

First, I'm slightly confused by the zfs create command there, which 
seems to mention a pool without a filesystem.  Something like zfs create 
osd.1/foo would create the filesystem in the pool osd.1, but just 'zfs 
create osd.1' mentions a filesystem but no filesystem, and:


# zfs create -o mountpoint=/tmp/foo osd.0
cannot create 'osd.0': missing dataset name

What I've done is

# zpool create -m/var/lib/ceph/osd/osd.0 osd.0 gpt/zd000 gpt/zd001

That creates a zfs (root) filesystem mounted at /var/lib/ceph/osd/osd.0:

# zfs list osd.0
NAMEUSED  AVAIL  REFER  MOUNTPOINT
osd.0   388K  7.02T88K  /var/lib/ceph/osd/osd.0

I'm following the setup instructions in 
, 
which (I believe) result in:


# ceph-authtool --create-keyring /tmp/ceph.mon.keyring \
--gen-key -n mon.0 --cap mon 'allow *'
# ceph-authtool --create-keyring 
/usr/local/etc/ceph/ceph.client.admin.keyring \

--gen-key -n client.admin --set-uid=0 --cap mon 'allow *' \
--cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'
# ceph-authtool /tmp/ceph.mon.keyring \
--import-keyring /usr/local/etc/ceph/ceph.client.admin.keyring
# monmaptool --create --add pochhammer  172.20.202.119 \
--fsid 08b41ee1-a080-11e8-9017-a0369fa1e52c /tmp/monmap
# mkdir /var/lib/ceph/mon/ceph-pochhammer
# chown ceph /var/lib/ceph/mon/ceph-pochhammer
# chmod 644 /tmp/ceph.mon.keyring
# sudo -u ceph ceph-mon --mkfs -i pochhammer \
--monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring

I'm experimenting here, and aiming to create a test monitor and a couple 
of OSDs on a single machine.


When I attempt to start the service, I get:

# service ceph start
=== mon.pochhammer ===
Starting Ceph mon.pochhammer on pochhammer...
2018-08-20 12:19:39.676389 9ee4480 -1 WARNING: 'mon addr' config option 
172.20.202.119:0/0 does not match monmap file

 continuing with monmap configuration
2018-08-20 12:19:39.677483 9ee4480 -1 auth: error reading file: 
/var/lib/ceph/mon/ceph-pochhammer/keyring: can't open 
/var/lib/ceph/mon/ceph-pochhammer/keyring: (2) No such file or directory
2018-08-20 12:19:39.677487 9ee4480 -1 mon.pochhammer@-1(probing) e0 
unable to load initial keyring 
/etc/ceph/ceph.mon.pochhammer.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,

2018-08-20 12:19:39.677492 9ee4480 -1 failed to initialize
failed: 'ulimit -n 32768;   /usr/local/bin/ceph-mon -i pochhammer 
--pid-file /var/run/ceph/mon.pochhammer.pid -c 
/usr/local/etc/ceph/ceph.conf --cluster ceph  --setuser ceph --setgroup 
ceph '


There are a couple of things about this that puzzle me.

1. I don't have a file /var/lib/ceph/mon/ceph-pochhammer/keyring.  After 
the setup instructions, I have a /tmp/ceph.mon.keyring and /tmp/monmap, 
but


# ls /var/lib/ceph/mon/ceph-pochhammer
donekv_backend  store.db/

Have I failed to copy something into place?  Similarly, I don't have any 
keyring copied into that directory.


2. I'm not sure what monmap file this is examining, as reported in the 
second file, presumably not the one in /tmp.  Also, is there a way to 
dump the contents of the/a monmap file? -- I can hexdump it, but that's 
not particularly illuminating.


I'm feeling a bit stuck at this point..., and feel I've surely missed 
some steps.


Best wishes,

Norman



My ceph.conf is minimally changed from the default:

[global]
fsid   = 08b41ee1-a080-11e8-9017-a0369fa1e52c
public network = 172.20.202.0/24
pid file   = /var/run/ceph/$name.pid
auth cluster required  = cephx
auth service required  = cephx
auth client required   = cephx
cephx cluster require signatures = true
cephx service require signatures = false
osd pool default size  = 2
osd pool default min size  = 2
osd pool default pg num= 350
osd pool default pgp num   = 350
[mon]
mon initial members= pochhammer
[mon.pochhammer]
  host = pochhammer
  mon addr = 172.20.202.119
[mds]
[osd]
[client]
rbd cache   = true
[client.radosgw.gateway]



--
Norman Gray  :  https://nxg.me.uk
SUPA School of Physics and Astronomy, University of Glasgow, UK
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] BlueStore sizing

[Paul Emmerich]

It will automatically spill over to the slower storage if necessary;
it's better to have some fast storage for the DB than just slow
storage.


Paul

2018-08-20 13:07 GMT+02:00 Harald Staub :
> As mentioned here recently, the sizing recommendations for BlueStore have
> been updated:
> http://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#sizing
>
> In our ceph cluster, we have some ratios that are much lower, like 20GB of
> SSD (WAL and DB) per 7TB of spinning space. This stems from the fact that it
> started much earlier, with sizing done with filestore based OSDs in mind
> (SSDs for journals), that were converted to BlueStore later.
>
> So SSD space is much lower than recommended, but much higher than needed for
> WAL only. Would it still be better to remove the DB from SSD? Or what
> problems could arise with our current setup? Up to now (several months), we
> do not see any disadvantages.
>
>  Harry
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Upgrade to Infernalis: failed to pick suitable auth object

[Kees Meijs]

The given PG is back online, phew...

Meanwhile, some OSDs still on Hammer seem to crash with errors alike:

> 2018-08-20 13:06:33.819569 7f8962b2f700 -1 osd/ReplicatedPG.cc: In
> function 'void ReplicatedPG::scan_range(int, int,
> PG::BackfillInterval*, ThreadPool::TPHandle&)' thread 7f8962b2f700
> time 2018-08-20 13:06:33.709922
> osd/ReplicatedPG.cc: 10115: FAILED assert(r >= 0)

Restarting the OSDs seems to work.

K.

On 20-08-18 13:14, Kees Meijs wrote:
> Bad news: I've got a PG stuck in down+peering now.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Upgrade to Infernalis: failed to pick suitable auth object

[Kees Meijs]

Bad news: I've got a PG stuck in down+peering now.

Please advice.

K.

On 20-08-18 12:12, Kees Meijs wrote:
> Thanks for your advice. My end goal is BlueStore so to upgrade to Jewel
> and then Luminous would be ideal.
>
> Currently all monitors are (succesfully) running Internalis, one OSD
> node is running Infernalis and all other OSD nodes have Hammer.
>
> I'll try freeing up one Infernalis OSD at first and see what'll happen.
> If it goes well I'll just (for now) give up all OSDs on the given node.
> If it works, I'll end up with Hammer OSDs only and Infernalis monitors.
>
> To be continued again!

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] BlueStore sizing

[Harald Staub]

As mentioned here recently, the sizing recommendations for BlueStore 
have been updated:

http://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#sizing

In our ceph cluster, we have some ratios that are much lower, like 20GB 
of SSD (WAL and DB) per 7TB of spinning space. This stems from the fact 
that it started much earlier, with sizing done with filestore based OSDs 
in mind (SSDs for journals), that were converted to BlueStore later.


So SSD space is much lower than recommended, but much higher than needed 
for WAL only. Would it still be better to remove the DB from SSD? Or 
what problems could arise with our current setup? Up to now (several 
months), we do not see any disadvantages.


 Harry
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ensure Hammer client compatibility

[Kees Meijs]

Good afternoon Cephers,

While I'm fixing our upgrade-semi-broken cluster (see thread Upgrade to
Infernalis: failed to pick suitable auth object) I'm wondering about
ensuring client compatibility.

My end goal is BlueStore (i.e. running Luminous) and unfortunately I'm
obliged to offer Hammer client compatibility.

Any pointers on how to ensure this configuration-wise?

Thanks!

Regards,
Kees

-- 
https://nefos.nl/contact

Nefos IT bv
Ambachtsweg 25 (industrienummer 4217)
5627 BZ Eindhoven
Nederland

KvK 66494931

/Aanwezig op maandag, dinsdag, woensdag en vrijdag/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Set existing pools to use hdd device class only

[David Turner]

All of the data moves because all of the crush IDs for the hosts and osds
changes when you configure a crush rule to only use SSDs or HDDs. Crush
creates shadow hosts and shadow osds in the crush map that only have each
class of osd.  So if you had node1 with osd.0 as an hdd and osd.1 as an
SSD, then you're crush roles using classes would use the shadow crush items
of node1-hdd with an odd with a different id for osd.0-hdd and the same for
the SSDs. This replaced the old behavior when you had to actually create a
dummy host in a second crush root that you created to do this yourself. And
you had to place all of the osds in their respective roots.

There is a thread from the ceph-large users ML that covered a way to do
this change without shifting data for an HDD only cluster.  Hopefully it
will be helpful for you.


httpists.ceph.com/pipermail/ceph-large-ceph.com/2018-April/000106.html

On Mon, Aug 20, 2018, 5:42 AM Enrico Kern  wrote:

> Hmm then is not really an option for me. Maybe someone from the devs can
> shed a light why it is doing migration as long you only have OSDs with the
> same class? I have a few Petabyte of Storage in each cluster. When it
> starts migrating everything again that will result in a super big
> performance bottleneck. My plan was to set the existing pools to use the
> new crush rule hdd only and add ssd osds later.
>
> On Mon, Aug 20, 2018 at 11:22 AM Marc Roos 
> wrote:
>
>>
>> I just recently did the same. Take into account that everything starts
>> migrating. How weird it maybe, I had hdd test cluster only and changed
>> the crush rule to having hdd. Took a few days, totally unnecessary as
>> far as I am concerned.
>>
>>
>>
>>
>> -Original Message-
>> From: Enrico Kern [mailto:enrico.k...@glispa.com]
>> Sent: maandag 20 augustus 2018 11:18
>> To: ceph-users@lists.ceph.com
>> Subject: [ceph-users] Set existing pools to use hdd device class only
>>
>> Hello,
>>
>> right now we have multiple HDD only clusters with ether filestore
>> journals on SSDs or on newer installations WAL etc. on SSD.
>>
>> I plan to extend our ceph clusters with SSDs to provide ssd only pools.
>> In luminous we have devices classes so that i should be able todo this
>> without editing around crush map.
>>
>> In the device class doc it says i can create "new" pools to use only
>> ssds as example:
>>
>> ceph osd crush rule create-replicated fast default host ssd
>>
>> what happens if i fire this on an existing pool but with hdd device
>> class? I wasnt able to test thi yet in our staging cluster and wanted to
>> ask whats the way todo this.
>>
>> I want to set an existing pool called volumes to only use osds with hdd
>> class. Right now all OSDs have hdds. So in theory it should not use
>> newly created SSD osds once i set them all up for hdd classes right?
>>
>> So for existing pool running:
>>
>> ceph osd crush rule create-replicated volume-hdd-only volumes default
>> hdd ceph osd pool set volumes crush_rule volume-hdd-only
>>
>>
>> should be the way to go right?
>>
>>
>> Regards,
>>
>> Enrico
>>
>> --
>>
>>
>> Enrico Kern
>>
>> VP IT Operations
>>
>>
>> enrico.k...@glispa.com
>> +49 (0) 30 555713017 / +49 (0) 152 26814501
>>
>> skype: flyersa
>> LinkedIn Profile 
>>
>>
>>   
>>
>>
>> Glispa GmbH | Berlin Office
>>
>> Stromstr. 11-17  
>> Berlin, Germany, 10551  
>>
>> Managing Director Din Karol-Gavish
>> Registered in Berlin
>> AG Charlottenburg |
>> <
>> https://maps.google.com/?q=Sonnenburger+Stra%C3%9Fe+73+10437+Berlin%C2%A0%7C+%3Chttps://maps.google.com/?q%3DSonnenburgerstra%25C3%259Fe%2B73%2B10437%2BBerlin%25C2%25A0%257C%25C2%25A0Germany%26entry%3Dgmail%26source%3Dg%3E%C2%A0Germany=gmail=g>
>>
>> HRB 114678B –
>>
>>
>>
>
> --
>
> *Enrico Kern*
> VP IT Operations
>
> enrico.k...@glispa.com
> +49 (0) 30 555713017 / +49 (0) 152 26814501
> skype: flyersa
> LinkedIn Profile 
>
>
>  
>
> *Glispa GmbH* | Berlin Office
> Stromstr. 11-17  
> Berlin, Germany, 10551  
>
> Managing Director Din Karol-Gavish
> Registered in Berlin
> AG Charlottenburg |
> 
>  HRB
> 114678B
> –
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Upgrade to Infernalis: failed to pick suitable auth object

[Kees Meijs]

Hi David,

Thanks for your advice. My end goal is BlueStore so to upgrade to Jewel
and then Luminous would be ideal.

Currently all monitors are (succesfully) running Internalis, one OSD
node is running Infernalis and all other OSD nodes have Hammer.

I'll try freeing up one Infernalis OSD at first and see what'll happen.
If it goes well I'll just (for now) give up all OSDs on the given node.
If it works, I'll end up with Hammer OSDs only and Infernalis monitors.

To be continued again!

Regards,
Kees

On 20-08-18 12:04, David Turner wrote:
> My suggestion would be to remove the osds and let the cluster recover
> from all of the other copies. I would deploy the node back to Hammer
> instead of Infernalis. Either that or remove these osds, let the
> cluster backfill, and then upgrade to Jewel, and then luminous, and
> maybe mimic if you're planning on making it to the newest LTS before
> adding the node back in. That way you could add them back in as
> bluestore (on either luminous or mimic) if that's a part of your plan.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Upgrade to Infernalis: failed to pick suitable auth object

[David Turner]

My suggestion would be to remove the osds and let the cluster recover from
all of the other copies. I would deploy the node back to Hammer instead of
Infernalis. Either that or remove these osds, let the cluster backfill, and
then upgrade to Jewel, and then luminous, and maybe mimic if you're
planning on making it to the newest LTS before adding the node back in.
That way you could add them back in as bluestore (on either luminous or
mimic) if that's a part of your plan.

On Mon, Aug 20, 2018, 5:55 AM Kees Meijs  wrote:

> Ehrm, that should of course be rebuilding. (I.e. removing the OSD,
> reformat, re-add.)
>
> On 20-08-18 11:51, Kees Meijs wrote:
> > Since there's temp in the name and we're running a 3-replica cluster,
> > I'm thinking of just reboiling the comprised OSDs.
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Upgrade to Infernalis: failed to pick suitable auth object

[Kees Meijs]

Ehrm, that should of course be rebuilding. (I.e. removing the OSD,
reformat, re-add.)

On 20-08-18 11:51, Kees Meijs wrote:
> Since there's temp in the name and we're running a 3-replica cluster,
> I'm thinking of just reboiling the comprised OSDs.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Tons of "cls_rgw.cc:3284: gc_iterate_entries end_key=" records in OSD logs

[David Turner]

I'm assuming you use RGW and that you have a GC pool for RGW. It also might
beat assumed that your GC pool only has 8 PGs.  Are any of those guesses
correct?

On Mon, Aug 20, 2018, 5:13 AM Jakub Jaszewski 
wrote:

> Issue tracker http://tracker.ceph.com/issues/23801.
> Still don't know why only particular OSDs write this information to log
> files.
>
> Jakub
>
> On Wed, Aug 8, 2018 at 12:02 PM Jakub Jaszewski 
> wrote:
>
>> Hi All, exactly the same story today, same 8 OSDs and a lot of garbage
>> collection objects to process
>>
>> Below is the number of "cls_rgw.cc:3284: gc_iterate_entries end_key="
>> entries per OSD log file
>> hostA:
>>   /var/log/ceph/ceph-osd.58.log
>>   1826467
>> hostB:
>>   /var/log/ceph/ceph-osd.88.log
>>   2924241
>> hostC:
>>   /var/log/ceph/ceph-osd.153.log
>>   581002
>>   /var/log/ceph/ceph-osd.164.log
>>   3278606
>> hostD:
>>   /var/log/ceph/ceph-osd.95.log
>>   1426963
>> hostE:
>>   /var/log/ceph/ceph-osd.4.log
>>   2716914
>>   /var/log/ceph/ceph-osd.53.log
>>   943749
>> hostF:
>>   /var/log/ceph/ceph-osd.172.log
>>   4085334
>>
>>
>> # radosgw-admin gc list --include-all|grep oid |wc -l
>> 302357
>> #
>>
>> Can anyone please explain what is going on ?
>>
>> Thanks!
>> Jakub
>>
>> On Tue, Aug 7, 2018 at 3:03 PM Jakub Jaszewski 
>> wrote:
>>
>>> Hi,
>>>
>>> 8 out of 192 OSDs in our cluster (version 12.2.5) write plenty of
>>> records like "cls_rgw.cc:3284: gc_iterate_entries end_key=" to the
>>> corresponding log files, e.g.
>>>
>>> 2018-08-07 04:34:06.000585 7fdd8f012700  0 
>>> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries
>>> end_key=1_01533616446.000580407
>>> 2018-08-07 04:34:06.001888 7fdd8f012700  0 
>>> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries
>>> end_key=1_01533616446.001886318
>>> 2018-08-07 04:34:06.003395 7fdd8f012700  0 
>>> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries
>>> end_key=1_01533616446.003390299
>>> 2018-08-07 04:34:06.005205 7fdd8f012700  0 
>>> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries
>>> end_key=1_01533616446.005200341
>>>
>>> # grep '2018-08-07 04:34:06' /var/log/ceph/ceph-osd.4.log |wc -l
>>> 712
>>> #
>>>
>>> At the same time there were like 500 000 expired garbage collection
>>> objects.
>>>
>>> Log level of OSD subsystem is set to default 1/5 across all OSDs.
>>>
>>> I wonder why only few OSDs record this information and is it something
>>> to be logged in log level = 1 or maybe higher?
>>> https://github.com/ceph/ceph/blob/v12.2.5/src/cls/rgw/cls_rgw.cc#L3284
>>>
>>> Thanks
>>> Jakub
>>>
>> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Upgrade to Infernalis: failed to pick suitable auth object

[Kees Meijs]

Hi again,

Over night some other PGs seem inconsistent as well after being deep
scrubbed.

All affected OSDs log similar errors like:

> log [ERR] : 3.13 soid -5/0013/temp_3.13_0_16175425_287/head:
> failed to pick suitable auth object

Since there's temp in the name and we're running a 3-replica cluster,
I'm thinking of just reboiling the comprised OSDs.

Any thoughts on this, can I do this safely?

Current status:

> 12 active+clean+inconsistent

Nota bene: it cannot be file ownership is the real culprit of this. Like
I mentioned earlier in this thread it might be the case for one or maybe
two OSDs but definitely not all.

Regards,
Kees

On 19-08-18 08:55, Kees Meijs wrote:
> I'll investigate further.


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Librados Keyring Issues

[David Turner]

It isn't possible in the config file. You have to do it via the rados
constructor. You came to the correct conclusion.

On Mon, Aug 20, 2018, 2:59 AM Benjamin Cherian 
wrote:

> Ok...after a bit more searching. I realized you can specify the username
> directly in the constructor of the "Rados" object. I'm still not entirely
> clear how one would do it through the config file, but this works for me as
> well.
>
> import rados
> cluster = rados.Rados(conffile="python_ceph.conf", rados_id="dms")
> cluster.connect() # No exception when using keyring containing key for dms
> user!
>
> Regards,
>
> Benjamin Cherian
>
> On Sun, Aug 19, 2018 at 9:55 PM, Benjamin Cherian <
> benjamin.cher...@gmail.com> wrote:
>
>> Hi David,
>>
>> Thanks for the reply...I had thought there might be something simple like
>> this, do you know what key I should use in the config file to specify the
>> user? I didn't see anything related to user specification in the
>> documentation.
>>
>> Thanks,
>> Ben
>>
>> On Sun, Aug 19, 2018 at 8:02 PM, David Turner 
>> wrote:
>>
>>> You are not specifying which user you are using. Your config file
>>> specifies the keyring, but it's still trying to use the default user admin.
>>> If you specify that in your python you'll be good to go.
>>>
>>> On Sun, Aug 19, 2018, 9:17 PM Benjamin Cherian <
>>> benjamin.cher...@gmail.com> wrote:
>>>
 Hi,

 I'm trying to write a simple test application using the Python 3 RADOS
 API. I've made a separate keyring for my application with the same
 permissions as the admin keyring, but I can't seem to use it to connect to
 my cluster. The only keyring that seems to work is the client.admin
 keyring. Does anyone have any experience with this issue?

 Cluster info:
 OS: Ubuntu 18.04.1
 Ceph: Mimic 13.2.1 (from Ceph repository)

 Attempting to connect to the cluster using python3-rados or
 python-rados results in the following error:

 >>> import rados
 >>> cluster = rados.Rados(conffile="python_ceph.conf")
 >>> cluster.connect()
 Traceback (most recent call last):
   File "", line 1, in 
   File "rados.pyx", line 895, in rados.Rados.connect
   File "rados.pyx", line 474, in rados.make_ex
 TypeError: InvalidArgumentError does not take keyword arguments

 Contents of python_ceph.conf:

 [global]
 cluster network = 0.0.0.0/0
 fsid = 518403a0-6b6f-42b8-be99-e58788bee5c2
 mon host = 
 mon initial members = 
 mon_allow_pool_delete = True
 osd crush chooseleaf type = 0
 public network = 0.0.0.0/0

 keyring = /etc/ceph/ceph.client.dms.keyring # Everything works ok if i
 use client.adming.keyring



 Output of ceph auth ls

 ...
 client.admin
 key: 
 caps: [mds] allow *
 caps: [mgr] allow *
 caps: [mon] allow *
 caps: [osd] allow *
 ...
 client.dms
 key: 
 caps: [mgr] allow *
 caps: [mon] allow *
 caps: [osd] allow *

 ...


 What's even more odd is that I can use client.dms keyring with the ceph
 command line program without issues...
 e.g., "ceph --user dms status" does not result in any errors has the
 same output as "ceph --user admin status"


 Does anyone have any thoughts on what could be causing this issue?


 Thanks,
 Ben
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

>>>
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] missing dependecy in ubuntu packages

[John Spray]

On Sun, Aug 19, 2018 at 9:21 PM Alfredo Daniel Rezinovsky
 wrote:
>
> both in ubuntu 16.04 and 18.04 ceph-mgr fail to starts when package
> python-routes is not installed

I guess you mean that the dashboard doesn't work, as opposed to the
whole ceph-mgr process not starting?  If it's the latter that's
worrying...

As far as I can tell, the dashboard doesn't actually use the routes
package itself.  I wonder if there is another dependency that uses
routes, and that dependency's packaging has a missing dependency on
routes?  If you look in the ceph-mgr log you should see a backtrace
somewhere that would show which piece of code is trying to use the
routes module.

John

> Some python packages are listed as dependencies for ceph-mgr but
> python-routes is missing and must be installed manually for ceph-mgr to
> work.
>
> --
> Alfredo Daniel Rezinovsky
> Director de Tecnologías de Información y Comunicaciones
> Facultad de Ingeniería - Universidad Nacional de Cuyo
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Set existing pools to use hdd device class only

[Enrico Kern]

Hmm then is not really an option for me. Maybe someone from the devs can
shed a light why it is doing migration as long you only have OSDs with the
same class? I have a few Petabyte of Storage in each cluster. When it
starts migrating everything again that will result in a super big
performance bottleneck. My plan was to set the existing pools to use the
new crush rule hdd only and add ssd osds later.

On Mon, Aug 20, 2018 at 11:22 AM Marc Roos  wrote:

>
> I just recently did the same. Take into account that everything starts
> migrating. How weird it maybe, I had hdd test cluster only and changed
> the crush rule to having hdd. Took a few days, totally unnecessary as
> far as I am concerned.
>
>
>
>
> -Original Message-
> From: Enrico Kern [mailto:enrico.k...@glispa.com]
> Sent: maandag 20 augustus 2018 11:18
> To: ceph-users@lists.ceph.com
> Subject: [ceph-users] Set existing pools to use hdd device class only
>
> Hello,
>
> right now we have multiple HDD only clusters with ether filestore
> journals on SSDs or on newer installations WAL etc. on SSD.
>
> I plan to extend our ceph clusters with SSDs to provide ssd only pools.
> In luminous we have devices classes so that i should be able todo this
> without editing around crush map.
>
> In the device class doc it says i can create "new" pools to use only
> ssds as example:
>
> ceph osd crush rule create-replicated fast default host ssd
>
> what happens if i fire this on an existing pool but with hdd device
> class? I wasnt able to test thi yet in our staging cluster and wanted to
> ask whats the way todo this.
>
> I want to set an existing pool called volumes to only use osds with hdd
> class. Right now all OSDs have hdds. So in theory it should not use
> newly created SSD osds once i set them all up for hdd classes right?
>
> So for existing pool running:
>
> ceph osd crush rule create-replicated volume-hdd-only volumes default
> hdd ceph osd pool set volumes crush_rule volume-hdd-only
>
>
> should be the way to go right?
>
>
> Regards,
>
> Enrico
>
> --
>
>
> Enrico Kern
>
> VP IT Operations
>
>
> enrico.k...@glispa.com
> +49 (0) 30 555713017 / +49 (0) 152 26814501
>
> skype: flyersa
> LinkedIn Profile 
>
>
>   
>
>
> Glispa GmbH | Berlin Office
>
> Stromstr. 11-17  
> Berlin, Germany, 10551  
>
> Managing Director Din Karol-Gavish
> Registered in Berlin
> AG Charlottenburg |
> <
> https://maps.google.com/?q=Sonnenburger+Stra%C3%9Fe+73+10437+Berlin%C2%A0%7C+%3Chttps://maps.google.com/?q%3DSonnenburgerstra%25C3%259Fe%2B73%2B10437%2BBerlin%25C2%25A0%257C%25C2%25A0Germany%26entry%3Dgmail%26source%3Dg%3E%C2%A0Germany=gmail=g>
>
> HRB 114678B –
>
>
>

-- 

*Enrico Kern*
VP IT Operations

enrico.k...@glispa.com
+49 (0) 30 555713017 / +49 (0) 152 26814501
skype: flyersa
LinkedIn Profile 


 

*Glispa GmbH* | Berlin Office
Stromstr. 11-17  
Berlin, Germany, 10551  

Managing Director Din Karol-Gavish
Registered in Berlin
AG Charlottenburg |

HRB
114678B
–
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Set existing pools to use hdd device class only

[Marc Roos]

 
I just recently did the same. Take into account that everything starts 
migrating. How weird it maybe, I had hdd test cluster only and changed 
the crush rule to having hdd. Took a few days, totally unnecessary as 
far as I am concerned.




-Original Message-
From: Enrico Kern [mailto:enrico.k...@glispa.com] 
Sent: maandag 20 augustus 2018 11:18
To: ceph-users@lists.ceph.com
Subject: [ceph-users] Set existing pools to use hdd device class only

Hello,

right now we have multiple HDD only clusters with ether filestore 
journals on SSDs or on newer installations WAL etc. on SSD.

I plan to extend our ceph clusters with SSDs to provide ssd only pools. 
In luminous we have devices classes so that i should be able todo this 
without editing around crush map.

In the device class doc it says i can create "new" pools to use only 
ssds as example:

ceph osd crush rule create-replicated fast default host ssd

what happens if i fire this on an existing pool but with hdd device 
class? I wasnt able to test thi yet in our staging cluster and wanted to 
ask whats the way todo this.

I want to set an existing pool called volumes to only use osds with hdd 
class. Right now all OSDs have hdds. So in theory it should not use 
newly created SSD osds once i set them all up for hdd classes right?

So for existing pool running:

ceph osd crush rule create-replicated volume-hdd-only volumes default 
hdd ceph osd pool set volumes crush_rule volume-hdd-only


should be the way to go right?


Regards,

Enrico

-- 


Enrico Kern

VP IT Operations


enrico.k...@glispa.com
+49 (0) 30 555713017 / +49 (0) 152 26814501

skype: flyersa
LinkedIn Profile  


   


Glispa GmbH | Berlin Office

Stromstr. 11-17  
Berlin, Germany, 10551   

Managing Director Din Karol-Gavish
Registered in Berlin
AG Charlottenburg | 

  
HRB 114678B –


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Set existing pools to use hdd device class only

[Enrico Kern]

Hello,

right now we have multiple HDD only clusters with ether filestore journals
on SSDs or on newer installations WAL etc. on SSD.

I plan to extend our ceph clusters with SSDs to provide ssd only pools. In
luminous we have devices classes so that i should be able todo this without
editing around crush map.

In the device class doc it says i can create "new" pools to use only ssds
as example:

ceph osd crush rule create-replicated fast default host ssd


what happens if i fire this on an existing pool but with hdd device class?
I wasnt able to test thi yet in our staging cluster and wanted to ask whats
the way todo this.

I want to set an existing pool called volumes to only use osds with hdd
class. Right now all OSDs have hdds. So in theory it should not use newly
created SSD osds once i set them all up for hdd classes right?

So for existing pool running:

ceph osd crush rule create-replicated volume-hdd-only volumes default hdd
ceph osd pool set volumes crush_rule volume-hdd-only

should be the way to go right?


Regards,

Enrico

-- 

*Enrico Kern*
VP IT Operations

enrico.k...@glispa.com
+49 (0) 30 555713017 / +49 (0) 152 26814501
skype: flyersa
LinkedIn Profile 


 

*Glispa GmbH* | Berlin Office
Stromstr. 11-17  
Berlin, Germany, 10551  

Managing Director Din Karol-Gavish
Registered in Berlin
AG Charlottenburg |

HRB
114678B
–
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Tons of "cls_rgw.cc:3284: gc_iterate_entries end_key=" records in OSD logs

[Jakub Jaszewski]

Issue tracker http://tracker.ceph.com/issues/23801.
Still don't know why only particular OSDs write this information to log
files.

Jakub

On Wed, Aug 8, 2018 at 12:02 PM Jakub Jaszewski 
wrote:

> Hi All, exactly the same story today, same 8 OSDs and a lot of garbage
> collection objects to process
>
> Below is the number of "cls_rgw.cc:3284: gc_iterate_entries end_key="
> entries per OSD log file
> hostA:
>   /var/log/ceph/ceph-osd.58.log
>   1826467
> hostB:
>   /var/log/ceph/ceph-osd.88.log
>   2924241
> hostC:
>   /var/log/ceph/ceph-osd.153.log
>   581002
>   /var/log/ceph/ceph-osd.164.log
>   3278606
> hostD:
>   /var/log/ceph/ceph-osd.95.log
>   1426963
> hostE:
>   /var/log/ceph/ceph-osd.4.log
>   2716914
>   /var/log/ceph/ceph-osd.53.log
>   943749
> hostF:
>   /var/log/ceph/ceph-osd.172.log
>   4085334
>
>
> # radosgw-admin gc list --include-all|grep oid |wc -l
> 302357
> #
>
> Can anyone please explain what is going on ?
>
> Thanks!
> Jakub
>
> On Tue, Aug 7, 2018 at 3:03 PM Jakub Jaszewski 
> wrote:
>
>> Hi,
>>
>> 8 out of 192 OSDs in our cluster (version 12.2.5) write plenty of records
>> like "cls_rgw.cc:3284: gc_iterate_entries end_key=" to the corresponding
>> log files, e.g.
>>
>> 2018-08-07 04:34:06.000585 7fdd8f012700  0 
>> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries
>> end_key=1_01533616446.000580407
>> 2018-08-07 04:34:06.001888 7fdd8f012700  0 
>> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries
>> end_key=1_01533616446.001886318
>> 2018-08-07 04:34:06.003395 7fdd8f012700  0 
>> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries
>> end_key=1_01533616446.003390299
>> 2018-08-07 04:34:06.005205 7fdd8f012700  0 
>> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries
>> end_key=1_01533616446.005200341
>>
>> # grep '2018-08-07 04:34:06' /var/log/ceph/ceph-osd.4.log |wc -l
>> 712
>> #
>>
>> At the same time there were like 500 000 expired garbage collection
>> objects.
>>
>> Log level of OSD subsystem is set to default 1/5 across all OSDs.
>>
>> I wonder why only few OSDs record this information and is it something to
>> be logged in log level = 1 or maybe higher?
>> https://github.com/ceph/ceph/blob/v12.2.5/src/cls/rgw/cls_rgw.cc#L3284
>>
>> Thanks
>> Jakub
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Invalid Object map without flags set

[Glen Baars]

Hello K,

We have found our issue – we were only fixing the main RDB image in our script 
rather than the snapshots. Working fine now.

Thanks for your help.
Kind regards,
Glen Baars
From: Konstantin Shalygin 
Sent: Friday, 17 August 2018 11:20 AM
To: ceph-users@lists.ceph.com; Glen Baars 
Subject: Re: [ceph-users] Invalid Object map without flags set


We are having issues with ensuring that object-map and fast-diff is working 
correctly. Most of the time when there is an invalid fast-diff map, the flag is 
set to correctly indicate this. We have a script that checks for this and 
rebuilds object maps as required. If we don't fix these, snapshot removal and 
rbd usage commands are too slow.



About 10% of the time when we issue an `rbd du` command we get the following 
situation.



warning: fast-diff map is invalid for 2d6b4502-f720-4c00-b4a4-8879e415f283 at 
18-D-2018-07-11:1109. 
operation may be slow.



When we check the `rbd info` it doesn't have any flags set.



[INFO]
{"name":"2d6b4502-f720-4c00-b4a4-8879e415f283","size":536870912000,"objects":128000,"order":22,"object_size":4194304,"block_name_prefix":"rbd_data.17928643c9869","format":2,"features":["layering","exclusive-lock","object-map","fast-diff","deep-flatten"],"flags":[],"create_timestamp":"Sat
 Apr 28 19:45:59 2018"}

[Feat]["layering","exclusive-lock","object-map","fast-diff","deep-flatten"]

[Flag][]



Is there another way to detect invalid object maps?



Ceph 12.2.7 - All Bluestore
As long as I remember, when object-map is bad you will see this flag on rbd 
info.



for e in `rbd ls replicated_rbd`; do echo "replicated_rbd/${e}"; rbd info 
replicated_rbd/${e} | grep "flag"; done





k

This e-mail is intended solely for the benefit of the addressee(s) and any 
other named recipient. It is confidential and may contain legally privileged or 
confidential information. If you are not the recipient, any use, distribution, 
disclosure or copying of this e-mail is prohibited. The confidentiality and 
legal privilege attached to this communication is not waived or lost by reason 
of the mistaken transmission or delivery to you. If you have received this 
e-mail in error, please notify us immediately.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Librados Keyring Issues

[Benjamin Cherian]

Ok...after a bit more searching. I realized you can specify the username
directly in the constructor of the "Rados" object. I'm still not entirely
clear how one would do it through the config file, but this works for me as
well.

import rados
cluster = rados.Rados(conffile="python_ceph.conf", rados_id="dms")
cluster.connect() # No exception when using keyring containing key for dms
user!

Regards,

Benjamin Cherian

On Sun, Aug 19, 2018 at 9:55 PM, Benjamin Cherian <
benjamin.cher...@gmail.com> wrote:

> Hi David,
>
> Thanks for the reply...I had thought there might be something simple like
> this, do you know what key I should use in the config file to specify the
> user? I didn't see anything related to user specification in the
> documentation.
>
> Thanks,
> Ben
>
> On Sun, Aug 19, 2018 at 8:02 PM, David Turner 
> wrote:
>
>> You are not specifying which user you are using. Your config file
>> specifies the keyring, but it's still trying to use the default user admin.
>> If you specify that in your python you'll be good to go.
>>
>> On Sun, Aug 19, 2018, 9:17 PM Benjamin Cherian <
>> benjamin.cher...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I'm trying to write a simple test application using the Python 3 RADOS
>>> API. I've made a separate keyring for my application with the same
>>> permissions as the admin keyring, but I can't seem to use it to connect to
>>> my cluster. The only keyring that seems to work is the client.admin
>>> keyring. Does anyone have any experience with this issue?
>>>
>>> Cluster info:
>>> OS: Ubuntu 18.04.1
>>> Ceph: Mimic 13.2.1 (from Ceph repository)
>>>
>>> Attempting to connect to the cluster using python3-rados or python-rados
>>> results in the following error:
>>>
>>> >>> import rados
>>> >>> cluster = rados.Rados(conffile="python_ceph.conf")
>>> >>> cluster.connect()
>>> Traceback (most recent call last):
>>>   File "", line 1, in 
>>>   File "rados.pyx", line 895, in rados.Rados.connect
>>>   File "rados.pyx", line 474, in rados.make_ex
>>> TypeError: InvalidArgumentError does not take keyword arguments
>>>
>>> Contents of python_ceph.conf:
>>>
>>> [global]
>>> cluster network = 0.0.0.0/0
>>> fsid = 518403a0-6b6f-42b8-be99-e58788bee5c2
>>> mon host = 
>>> mon initial members = 
>>> mon_allow_pool_delete = True
>>> osd crush chooseleaf type = 0
>>> public network = 0.0.0.0/0
>>>
>>> keyring = /etc/ceph/ceph.client.dms.keyring # Everything works ok if i
>>> use client.adming.keyring
>>>
>>>
>>>
>>> Output of ceph auth ls
>>>
>>> ...
>>> client.admin
>>> key: 
>>> caps: [mds] allow *
>>> caps: [mgr] allow *
>>> caps: [mon] allow *
>>> caps: [osd] allow *
>>> ...
>>> client.dms
>>> key: 
>>> caps: [mgr] allow *
>>> caps: [mon] allow *
>>> caps: [osd] allow *
>>>
>>> ...
>>>
>>>
>>> What's even more odd is that I can use client.dms keyring with the ceph
>>> command line program without issues...
>>> e.g., "ceph --user dms status" does not result in any errors has the
>>> same output as "ceph --user admin status"
>>>
>>>
>>> Does anyone have any thoughts on what could be causing this issue?
>>>
>>>
>>> Thanks,
>>> Ben
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] packages names for ubuntu/debian

[Bastiaan Visser]

you should only use the 18.04 repo in 18.04, and remove the 16.04 repo.

use:
https://download.ceph.com/debian-luminous bionic main

- Bastiaan

- Original Message -
From: "Alfredo Daniel Rezinovsky" 
To: "ceph-users" 
Sent: Sunday, August 19, 2018 10:15:00 PM
Subject: [ceph-users] packages names for ubuntu/debian

Last packages for ubuntu 16.04 are version 13.2.1-1xenial
while last packages for ubuntu 18.04 are 13.2.1-1bionic

I recently upgraded from ubuntu 16 to 18 and the ceph packages stayed in xenial 
because alphabetically xenial > bionic.

I had to set the piining to force the upgrade to bionic (Which was trated as a 
downgrade)

In Ubuntu maling lists they said those packages are "wrongly versioned"

I think the names should be 13.2.1-1ubuntu16.04-xenial and 
13.2.1.ubuntu18.04-bionic.

-- 
Alfredo Daniel Rezinovsky
Director de Tecnologías de Información y Comunicaciones
Facultad de Ingeniería - Universidad Nacional de Cuyo

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com