[ceph-users] Re: dashboard not working

[Robert Sander]

Hi,

On 17.09.19 01:10, solarflow99 wrote:
> I have mimic installed and for some reason the dashboard isn't showing
> up.  I see which mon is listed as active for "mgr", the module is
> enabled, but nothing is listening on port 8080:
> 
> # ceph mgr module ls
> {
>     "enabled_modules": [
>         "dashboard",
>         "iostat",
>         "status"
> 
> 
> tcp        0      0 10.8.152.4:6800         
> 0.0.0.0:*               LISTEN      69049/ceph-mgr

Check with "ceph mgr services" which URL should be used.

Sometimes ceph-mgr needs a restart to activate a module.

Regards
-- 
Robert Sander
Heinlein Support GmbH
Schwedter Str. 8/9b, 10119 Berlin

https://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Amtsgericht Berlin-Charlottenburg - HRB 93818 B
Geschäftsführer: Peer Heinlein - Sitz: Berlin



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Nautilus: pg_autoscaler causes mon slow ops

[Eugen Block]

Hi everyone,

we upgraded our production cluster just recently to version:

ceph version 14.2.3-349-g7b1552ea82  
(7b1552ea827cf5167b6edbba96dd1c4a9dc16937) nautilus (stable)


We then activated pg_autoscaler for two pools that had a bad pg_num  
and the result is satisfying.
However, after the rebalance finished the cluster became laggy. We  
noticed that two out of three MONs had a much higher CPU usage than  
usual, according to `top` the MON processes consumed more than 100%.  
Restarting the MON services and disabling pg_autoscaler resolved the  
issue. I've read that the balancer module can cause a higher load on  
the MGR daemon, is this somehow related?



Another thing to mention is the confusing calculation of the  
autoscaler. After the pg numbers had been corrected we got the warning  
about overcommitted pools:



1 subtrees have overcommitted pool target_size_bytes
1 subtrees have overcommitted pool target_size_ratio


The images pool was responsible for that. The confusing part was that  
sometimes autoscale-status displayed the size of that pool with more  
than 14 TB:


ceph osd pool autoscale-status
 POOL   SIZE  TARGET SIZE  RATE  RAW CAPACITY   RATIO  
 TARGET RATIO  BIAS  PG_NUM  NEW PG_NUM  AUTOSCALE  
 images   14399G    3.0    33713G  1.2813  
1.0 128  on



And a couple of minutes later the pool suddenly only had around 4 TB of data:

ceph osd pool autoscale-status
 POOL   SIZE  TARGET SIZE  RATE  RAW CAPACITY   RATIO  
 TARGET RATIO  BIAS  PG_NUM  NEW PG_NUM  AUTOSCALE  
 images    4112G    3.0    33713G  0.3659  
1.0 128  on  



There seems to be some kind of inconsistency here. The actual used  
storage of this pool according to `ceph df` is:


POOLS:
POOLID STORED  OBJECTS USED 
%USED MAX AVAIL
images   1 4.1 TiB   1.01M  12 TiB  
49.73   4.1 TiB


Has anyone experienced something similar? Are these known issues?

Regards,
Eugen
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: dashboard not working

[Thomas]

Hi,
you should run
 ceph dashboard create-self-signed-cert
as documented here  (To
get the dashboard up and running quickly, you can generate and install a
self-signed certificate using the following built-in command).

Regards
Thomas

Am 17.09.2019 um 09:12 schrieb Robert Sander:
> Hi,
>
> On 17.09.19 01:10, solarflow99 wrote:
>> I have mimic installed and for some reason the dashboard isn't showing
>> up.  I see which mon is listed as active for "mgr", the module is
>> enabled, but nothing is listening on port 8080:
>>
>> # ceph mgr module ls
>> {
>>     "enabled_modules": [
>>         "dashboard",
>>         "iostat",
>>         "status"
>>
>>
>> tcp        0      0 10.8.152.4:6800         
>> 0.0.0.0:*               LISTEN      69049/ceph-mgr
> Check with "ceph mgr services" which URL should be used.
>
> Sometimes ceph-mgr needs a restart to activate a module.
>
> Regards
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: dashboard not working

[solarflow99]

I'm not trying to use SSL.  It seems strange how the mgr log doesn't even
mention anything about dashboard

On Tue, Sep 17, 2019 at 12:19 AM Thomas <74cmo...@gmail.com> wrote:

> Hi,
> you should run
>  ceph dashboard create-self-signed-cert
> as documented here  (To
> get the dashboard up and running quickly, you can generate and install a
> self-signed certificate using the following built-in command).
>
> Regards
> Thomas
>
> Am 17.09.2019 um 09:12 schrieb Robert Sander:
> > Hi,
> >
> > On 17.09.19 01:10, solarflow99 wrote:
> >> I have mimic installed and for some reason the dashboard isn't showing
> >> up.  I see which mon is listed as active for "mgr", the module is
> >> enabled, but nothing is listening on port 8080:
> >>
> >> # ceph mgr module ls
> >> {
> >> "enabled_modules": [
> >> "dashboard",
> >> "iostat",
> >> "status"
> >>
> >>
> >> tcp0  0 10.8.152.4:6800 
> >> 0.0.0.0:*   LISTEN  69049/ceph-mgr
> > Check with "ceph mgr services" which URL should be used.
> >
> > Sometimes ceph-mgr needs a restart to activate a module.
> >
> > Regards
> >
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 14.2.4 Packages Avaliable

[Daniel Baumann]

On 9/17/19 8:46 AM, Ronny Aasen wrote:
> Never install packages until there is an announcement.

which defeats the purpose of having the repositories in the first place.

> IIRC developers have asked if anyone have experience with running repos
> that could assist in improving the rollout of releases since this have
> been a recurring issue.

I'm happy to help (dan...@debian.org, dnl on freenode).

Regards,
Daniel
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: dashboard not working

[Robert Sander]

On 17.09.19 09:21, solarflow99 wrote:
> I'm not trying to use SSL.  It seems strange how the mgr log doesn't
> even mention anything about dashboard

If you do not want to use SSL, run "ceph config set mgr
mgr/dashboard/ssl false" to disable it. Otherwise Dashboard may not
start without a certificate.

Regards
-- 
Robert Sander
Heinlein Support GmbH
Schwedter Str. 8/9b, 10119 Berlin

https://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Amtsgericht Berlin-Charlottenburg - HRB 93818 B
Geschäftsführer: Peer Heinlein - Sitz: Berlin



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: dashboard not working

[Thomas]

In my case the dashboard is starting & running w/o any issues.
And I didn't disable ssl in config.


Am 17.09.2019 um 09:23 schrieb Robert Sander:
> On 17.09.19 09:21, solarflow99 wrote:
>> I'm not trying to use SSL.  It seems strange how the mgr log doesn't
>> even mention anything about dashboard
> If you do not want to use SSL, run "ceph config set mgr
> mgr/dashboard/ssl false" to disable it. Otherwise Dashboard may not
> start without a certificate.
>
> Regards
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 14.2.4 Packages Avaliable

[Fyodor Ustinov]

Hi!

Fine! Maybe you know what a new user should do? Which does not yet have a local 
copy of the repository, and is now trying to install the latest version?

- Original Message -
> From: "Ronny Aasen" 
> To: ceph-users@ceph.io
> Sent: Tuesday, 17 September, 2019 09:46:03
> Subject: [ceph-users] Re: 14.2.4 Packages Avaliable

> On 17.09.2019 06:54, Ashley Merrick wrote:
>> Have just noticed their is packages available for 14.2.4..
>> 
>> I know with the whole 14.2.3 release and the notes not going out to a
>> good day or so later.. but this is not long after the 14.2.3 release..?
>> 
>> Was this release even meant to have come out? Makes it difficult for
>> people installing a new node if they can't reply on the current "stable"
>> packages that apt/yum will give them.
> 
> 
> Never install packages until there is an announcement.
> 
> IIRC developers have asked if anyone have experience with running repos
> that could assist in improving the rollout of releases since this have
> been a recurring issue.
> 
> 
> If you need to do installs all the time, and can not postpone until the
> repo settle. Consider rsyncing the repo after a release, and use that
> for installs.
> 
> kind regards
> Ronny
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 14.2.4 Packages Avaliable

[Bastiaan Visser]

Depends on the OS you are using. 
Version pinning is the search keyword you are looking for

Hi!

Fine! Maybe you know what a new user should do? Which does not yet have a local 
copy of the repository, and is now trying to install the latest version?

- Original Message -
> From: "Ronny Aasen" 
> To: ceph-users@ceph.io
> Sent: Tuesday, 17 September, 2019 09:46:03
> Subject: [ceph-users] Re: 14.2.4 Packages Avaliable

> On 17.09.2019 06:54, Ashley Merrick wrote:
>> Have just noticed their is packages available for 14.2.4..
>> 
>> I know with the whole 14.2.3 release and the notes not going out to a
>> good day or so later.. but this is not long after the 14.2.3 release..?
>> 
>> Was this release even meant to have come out? Makes it difficult for
>> people installing a new node if they can't reply on the current "stable"
>> packages that apt/yum will give them.
> 
> 
> Never install packages until there is an announcement.
> 
> IIRC developers have asked if anyone have experience with running repos
> that could assist in improving the rollout of releases since this have
> been a recurring issue.
> 
> 
> If you need to do installs all the time, and can not postpone until the
> repo settle. Consider rsyncing the repo after a release, and use that
> for installs.
> 
> kind regards
> Ronny
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 14.2.4 Packages Avaliable

[Wido den Hollander]

On 9/17/19 8:46 AM, Ronny Aasen wrote:
> On 17.09.2019 06:54, Ashley Merrick wrote:
>> Have just noticed their is packages available for 14.2.4..
>>
>> I know with the whole 14.2.3 release and the notes not going out to a
>> good day or so later.. but this is not long after the 14.2.3 release..?
>>
>> Was this release even meant to have come out? Makes it difficult for
>> people installing a new node if they can't reply on the current
>> "stable" packages that apt/yum will give them.
> 
> 
> Never install packages until there is an announcement.
> 

But if a user runs "yum update" or "apt upgrade" one should be able to
trust download.ceph.com

We can't assume that every user does version pinning.

> IIRC developers have asked if anyone have experience with running repos
> that could assist in improving the rollout of releases since this have
> been a recurring issue.
> 

That would be great! But it seems right now that packages are pushed,
but the Release Notes haven't been generated. This happened a couple of
times now.

Wido

> 
> If you need to do installs all the time, and can not postpone until the
> repo settle. Consider rsyncing the repo after a release, and use that
> for installs.
> 
> kind regards
> Ronny
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 14.2.4 Packages Avaliable

[Fyodor Ustinov]

Hi!

No. I looking information on https://ceph.io/ why I should not install version 
14.2.4, although it is in the repository and how it differs from 14.2.3

- Original Message -
> From: "Bastiaan Visser" 
> To: ceph-users@ceph.io
> Sent: Tuesday, 17 September, 2019 10:52:54
> Subject: [ceph-users] Re: 14.2.4 Packages Avaliable

> Depends on the OS you are using.
> Version pinning is the search keyword you are looking for
> 
> Hi!
> 
> Fine! Maybe you know what a new user should do? Which does not yet have a 
> local
> copy of the repository, and is now trying to install the latest version?
> 
> - Original Message -
>> From: "Ronny Aasen" 
>> To: ceph-users@ceph.io
>> Sent: Tuesday, 17 September, 2019 09:46:03
>> Subject: [ceph-users] Re: 14.2.4 Packages Avaliable
> 
>> On 17.09.2019 06:54, Ashley Merrick wrote:
>>> Have just noticed their is packages available for 14.2.4..
>>> 
>>> I know with the whole 14.2.3 release and the notes not going out to a
>>> good day or so later.. but this is not long after the 14.2.3 release..?
>>> 
>>> Was this release even meant to have come out? Makes it difficult for
>>> people installing a new node if they can't reply on the current "stable"
>>> packages that apt/yum will give them.
>> 
>> 
>> Never install packages until there is an announcement.
>> 
>> IIRC developers have asked if anyone have experience with running repos
>> that could assist in improving the rollout of releases since this have
>> been a recurring issue.
>> 
>> 
>> If you need to do installs all the time, and can not postpone until the
>> repo settle. Consider rsyncing the repo after a release, and use that
>> for installs.
>> 
>> kind regards
>> Ronny
>> 
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 14.2.4 Packages Avaliable

[Wido den Hollander]

On 9/17/19 6:54 AM, Ashley Merrick wrote:
> Have just noticed their is packages available for 14.2.4..
> 
> I know with the whole 14.2.3 release and the notes not going out to a
> good day or so later.. but this is not long after the 14.2.3 release..?
> 

It seems that the release is intentional. See this commit:
https://github.com/ceph/ceph/commit/cf78989688038a55b029e2535ae9fe58479fb369

It's from Friday the 13th (who releases on a Friday the 13th ;-) ?)
September.

Wido

> Was this release even meant to have come out? Makes it difficult for
> people installing a new node if they can't reply on the current "stable"
> packages that apt/yum will give them.
> 
> 
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
> 
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 14.2.4 Packages Avaliable

[Yoann Moulin]

Hello,

>> Have just noticed their is packages available for 14.2.4..
>>
>> I know with the whole 14.2.3 release and the notes not going out to a good 
>> day or so later.. but this is not long after the 14.2.3 release..?
>>
>> Was this release even meant to have come out? Makes it difficult for people 
>> installing a new node if they can't reply on the current "stable"
>> packages that apt/yum will give them.
> 
> 
> Never install packages until there is an announcement.
> 
> IIRC developers have asked if anyone have experience with running repos that 
> could assist in improving the rollout of releases since this have
> been a recurring issue.
> 
> If you need to do installs all the time, and can not postpone until the repo 
> settle. Consider rsyncing the repo after a release, and use that
> for installs.

That does not make sense. If a package is available on the repo and can be 
installed or update directly with apt or yum, it has to be the final
release package, any other statement must be flagged as RC or beta. Why don't 
you add RC flags on this package if it's not ready to publish?

I plan to install a new cluster with ceph-ansible, I don't have to take care of 
the release number as soon as it's the latest package available
on the official stable repo. Even on a short period, an « almost ready but not  
completely full tested » packages can really have an impact on
production servers, especially if not announce previously.

Best regards,

-- 
Yoann Moulin
EPFL IC-IT
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: dashboard not working

[Lenz Grimmer]

On 9/17/19 9:21 AM, solarflow99 wrote:

> I'm not trying to use SSL.  It seems strange how the mgr log doesn't 
> even mention anything about dashboard

Did you try running "ceph mgr services"? Does it show the dashboard as
running? Did you check the Mgr of the current active Mgr?

Lenz

-- 
SUSE Software Solutions Germany GmbH - Maxfeldstr. 5 - 90409 Nuernberg
GF: Felix Imendörffer, HRB 247165 (AG Nürnberg)



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: KRBD use Luminous upmap feature.Which version of the kernel should i ues?

[Ilya Dryomov]

On Tue, Sep 17, 2019 at 8:54 AM 潘东元  wrote:
>
> Thank you for your reply.
> so,i would like to verify this problem. i create a new VM as a
> client,it is kernel version:
> [root@localhost ~]# uname -a
> Linux localhost.localdomain 5.2.9-200.fc30.x86_64 #1 SMP Fri Aug 16
> 21:37:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
>
> First of all,use command:ceph features in my cluster
> [root@node-1 ~]# ceph features
> {
> "mon": {
> "group": {
> "features": "0x3ffddff8eeacfffb",
> "release": "luminous",
> "num": 3
> }
> },
> "osd": {
> "group": {
> "features": "0x3ffddff8eeacfffb",
> "release": "luminous",
> "num": 12
> }
> },
> "client": {
> "group": {
> "features": "0x3ffddff8eeacfffb",
> "release": "luminous",
> "num": 7
> }
> }
> }
>
> now, i have no jewel client.And then,i map a rbd image to new VM
> [root@localhost ~]# rbd map test
> /dev/rbd0
> map successful!
> new,use ceph features
> [root@node-1 ~]# ceph features
> {
> "mon": {
> "group": {
> "features": "0x3ffddff8eeacfffb",
> "release": "luminous",
> "num": 3
> }
> },
> "osd": {
> "group": {
> "features": "0x3ffddff8eeacfffb",
> "release": "luminous",
> "num": 12
> }
> },
> "client": {
> "group": {
> "features": "0x27018fb86aa42ada",
> "release": "jewel",
> "num": 1
> },
> "group": {
> "features": "0x3ffddff8eeacfffb",
> "release": "luminous",
> "num": 7
> }
> }
> }
> I have a jewel client.It is not an expectation.
> why? is it means i still can not use upmap feature?

You can.  The kernel client reports itself as jewel due to a technical
issue, fixed in kernel 5.3.  All luminous features are fully supported,
all you need to do is "ceph osd set-require-min-compat-client luminous"
to allow them to be used.

Note that if you actually enable upmap, it will prevent older clients
from connecting, so you will no longer be able to use pre-luminous and
pre-4.13 (RHEL/CentOS 7.5) kernels.

Thanks,

Ilya
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 14.2.4 Packages Avaliable

[Paul Emmerich]

14.2.4 is a bug-fix release for https://tracker.ceph.com/issues/41660

There are no other changes beside this fix

-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90

On Tue, Sep 17, 2019 at 10:34 AM Yoann Moulin  wrote:
>
> Hello,
>
> >> Have just noticed their is packages available for 14.2.4..
> >>
> >> I know with the whole 14.2.3 release and the notes not going out to a good 
> >> day or so later.. but this is not long after the 14.2.3 release..?
> >>
> >> Was this release even meant to have come out? Makes it difficult for 
> >> people installing a new node if they can't reply on the current "stable"
> >> packages that apt/yum will give them.
> >
> >
> > Never install packages until there is an announcement.
> >
> > IIRC developers have asked if anyone have experience with running repos 
> > that could assist in improving the rollout of releases since this have
> > been a recurring issue.
> >
> > If you need to do installs all the time, and can not postpone until the 
> > repo settle. Consider rsyncing the repo after a release, and use that
> > for installs.
>
> That does not make sense. If a package is available on the repo and can be 
> installed or update directly with apt or yum, it has to be the final
> release package, any other statement must be flagged as RC or beta. Why don't 
> you add RC flags on this package if it's not ready to publish?
>
> I plan to install a new cluster with ceph-ansible, I don't have to take care 
> of the release number as soon as it's the latest package available
> on the official stable repo. Even on a short period, an « almost ready but 
> not  completely full tested » packages can really have an impact on
> production servers, especially if not announce previously.
>
> Best regards,
>
> --
> Yoann Moulin
> EPFL IC-IT
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 14.2.4 Packages Avaliable

[Yoann Moulin]

Hello,

 Have just noticed their is packages available for 14.2.4..

 I know with the whole 14.2.3 release and the notes not going out to a good 
 day or so later.. but this is not long after the 14.2.3 release..?

 Was this release even meant to have come out? Makes it difficult for 
 people installing a new node if they can't reply on the current "stable"
 packages that apt/yum will give them.
>>>
>>> Never install packages until there is an announcement.
>>>
>>> IIRC developers have asked if anyone have experience with running repos 
>>> that could assist in improving the rollout of releases since this have
>>> been a recurring issue.
>>>
>>> If you need to do installs all the time, and can not postpone until the 
>>> repo settle. Consider rsyncing the repo after a release, and use that
>>> for installs.
>>
>> That does not make sense. If a package is available on the repo and can be 
>> installed or update directly with apt or yum, it has to be the final
>> release package, any other statement must be flagged as RC or beta. Why 
>> don't you add RC flags on this package if it's not ready to publish?
>>
>> I plan to install a new cluster with ceph-ansible, I don't have to take care 
>> of the release number as soon as it's the latest package available
>> on the official stable repo. Even on a short period, an « almost ready but 
>> not  completely full tested » packages can really have an impact on
>> production servers, especially if not announce previously.
>
> 14.2.4 is a bug-fix release for https://tracker.ceph.com/issues/41660
> 
> There are no other changes beside this fix

My reaction was not on this specific release but with this sentence :  « Never 
install packages until there is an announcement. » And also with
this one : « If you need to do installs all the time, and can not postpone 
until the repo settle. Consider rsyncing the repo after a release,
and use that for installs. » This sound crazy to me.

Best regards,

-- 
Yoann Moulin
EPFL IC-IT
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: KRBD use Luminous upmap feature.Which version of the kernel should i ues?

[潘东元]

Yes,i went on with the test.
1、
# ceph osd getmap -o om
got osdmap epoch 276

2、pool id 2 is my rbd pool
# osdmaptool om --test-map-pgs --pool 2
osdmaptool: osdmap file 'om'
pool 2 pg_num 64
#osdcountfirstprimaryc wtwt
osd.020550.31251
osd.127880.31251
osd.217770.31251
osd.422660.31251
osd.517660.31251
osd.625990.31251
osd.82111110.31251
osd.915660.31251
osd.1028660.31251
 in 12
 avg 16 stddev 9.97497 (0.623436x) (expected 3.82971 0.239357x))
 min osd.9 15
 max osd.10 28
size 00
size 10
size 20
size 364

3、
# osdmaptool om --upmap-pool rbd --upmap out --upmap-save out
osdmaptool: osdmap file 'om'
writing upmap command output to: out
checking for upmap cleanups
upmap, max-count 100, max deviation 0.01
 limiting to pools rbd (2)
osdmaptool: writing epoch 278 to om

# cat out
ceph osd pg-upmap-items 2.1 6 5
ceph osd pg-upmap-items 2.3 10 9
ceph osd pg-upmap-items 2.17 6 5
ceph osd pg-upmap-items 2.19 10 9
ceph osd pg-upmap-items 2.1a 10 9
ceph osd pg-upmap-items 2.1c 10 9
ceph osd pg-upmap-items 2.1e 6 5
ceph osd pg-upmap-items 2.1f 10 9
ceph osd pg-upmap-items 2.29 1 2
ceph osd pg-upmap-items 2.35 1 0
ceph osd pg-upmap-items 2.3a 6 5
ceph osd pg-upmap-items 2.3c 1 2
ceph osd pg-upmap-items 2.3d 1 2
ceph osd pg-upmap-items 2.3e 10 9
ceph osd pg-upmap-items 2.3f 1 2

4、
# ceph osd set-require-min-compat-client luminous
set require_min_compat_client to luminous

5、
# source out
set 2.1 pg_upmap_items mapping to [6->5]
set 2.3 pg_upmap_items mapping to [10->9]
set 2.17 pg_upmap_items mapping to [6->5]
set 2.19 pg_upmap_items mapping to [10->9]
set 2.1a pg_upmap_items mapping to [10->9]
set 2.1c pg_upmap_items mapping to [10->9]
set 2.1e pg_upmap_items mapping to [6->5]
set 2.1f pg_upmap_items mapping to [10->9]
set 2.29 pg_upmap_items mapping to [1->2]
set 2.35 pg_upmap_items mapping to [1->0]
set 2.3a pg_upmap_items mapping to [6->5]
set 2.3c pg_upmap_items mapping to [1->2]
set 2.3d pg_upmap_items mapping to [1->2]
set 2.3e pg_upmap_items mapping to [10->9]
set 2.3f pg_upmap_items mapping to [1->2]

6、
# osdmaptool om1 --test-map-pgs --pool 2
osdmaptool: osdmap file 'om1'
pool 2 pg_num 64
#osdcountfirstprimaryc wtwt
osd.021660.31251
osd.122660.31251
osd.221880.31251
osd.30000.08007811
osd.422660.31251
osd.521990.31251
osd.621660.31251
osd.70000.08007811
osd.82111110.31251
osd.921880.31251
osd.1022440.31251
osd.110000.08007811
 in 12
 avg 16 stddev 9.24662 (0.577914x) (expected 3.82971 0.239357x))
 min osd.0 21
 max osd.1 22
size 00
size 10
size 20
size 364

now,my rbd pool's pg perfect distribution.I try to map rbd image to new VM:
[root@localhost ~]# rbd map test
/dev/rbd0

[root@localhost ~]# mount /dev/rbd0 /root/gyt/
[root@localhost ~]# lsblk
NAMEMAJ:MIN RM SIZE RO TYPE MOUNTPOINT
rbd0251:00   1G  0 disk /root/gyt
vda 252:00  40G  0 disk
|-vda1  252:10   1G  0 part /boot
`-vda2  252:20  39G  0 part
  |-fedora_localhost--live-root 253:00  35G  0 lvm  /
  `-fedora_localhost--live-swap 253:10   4G  0 lvm  [SWAP]

[root@localhost ~]# rbd unmap test

ok,can be used nornally.

Thanks!

Ilya Dryomov  于2019年9月17日周二 下午5:10写道：
>
> On Tue, Sep 17, 2019 at 8:54 AM 潘东元  wrote:
> >
> > Thank you for your reply.
> > so,i would like to verify this problem. i create a new VM as a
> > client,it is kernel version:
> > [root@localhost ~]# uname -a
> > Linux localhost.localdomain 5.2.9-200.fc30.x86_64 #1 SMP Fri Aug 16
> > 21:37:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
> >
> > First of all,use command:ceph features in my cluster
> > [root@node-1 ~]# ceph features
> > {
> > "mon": {
> > "group": {
> > "features": "0x3ffddff8eeacfffb",
> > "release": "luminous",
> > "num": 3
> > }
> > },
> > "osd": {
> > "group": {
> > "features": "0x3ffddff8eeacfffb",
> > "release": "luminous",
> > "num": 12
> > }
> > },
> > "client": {
> > "group": {
> > "features": "0x3ffddff8eeacfffb",
> > "release": "luminous",
> > "num": 7
> > }
> > }
> > }
> >
> > now, i have no jewel client.And then,i map a rbd image to new VM
> > [root@localhost ~]# rbd map test
> > /dev/rbd0
> > map successful!
> > new,use ceph features
> > [root@node-1 ~]# ceph features
> > {
> > "mon": {
> > "group": {
> > "features": "0x3ffddff8

[ceph-users] Re: 14.2.4 Packages Avaliable

[Janne Johansson]

Den tis 17 sep. 2019 kl 12:52 skrev Yoann Moulin :

> Hello,
>
> >>> Never install packages until there is an announcement.
> >>>
>
> My reaction was not on this specific release but with this sentence :  «
> Never install packages until there is an announcement. » And also with
> this one : « If you need to do installs all the time, and can not postpone
> until the repo settle. Consider rsyncing the repo after a release,
> and use that for installs. » This sound crazy to me.
>

I would have to agree with this. If these statements are true:
1) devs say: don't install unless there is an announcement out
2) devs have previously dropped new releases before posting the
announcement and told people it was a mistake to "yum upgrade"/"apt
upgrade" before announcement

then the solution seems very simple, do NOT populate the repos with
packages that your users shouldn't install until release notes are out.

I am well aware of that 100s of things around packaging and testing a
release is hard, times the amount of platforms and arches you support, but
to the untrained eye, not pushing packages out seems like quite a binary
thing. Either you press whatever button there is that pushes the files out,
or you don't press it.

If there is a reason for people to not run packages until a certain
condition is true, then don't press that hypothetical button until you get
convinced that the condition is.

Among all other things that are hard about releases, that can't be the
worst?

-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] New lines "choose_args" in crush map

[Thomas]

Hi! I recently noticed new lines in my crush map. Here's an extract of
it: # choose_args choose_args 18446744073709551615 { { bucket_id -1
weight_set [ [ 33.839 34.159 34.041 34.768 1.810 1.832 ] ] } { bucket_id
-2 weight_set [ [ 28.019 28.339 28.221 28.948 1.810 1.832 ] ] } {
bucket_id -3 weight_set [ [ 2.910 2.910 1.663 1.605 1.646 1.646 1.663
1.694 1.671 1.663 1.605 1.678 1.617 1.656 1.678 1.588 1.653 1.635 1.660
] ] } { bucket_id -4 weight_set [ [ 1.663 1.605 1.646 1.646 1.663 1.694
1.671 1.663 1.605 1.678 1.617 1.656 1.678 1.588 1.653 1.635 1.660 ] ] }
{ bucket_id -5 weight_set [ [ 2.910 2.910 ] ] } { bucket_id -6
weight_set [ [ 5.820 5.820 5.820 5.820 0.000 0.000 ] ] } [...] Question:
What is writing these lines into crush map? What is the function behind
these lines? THX Thomas
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 14.2.4 Packages Avaliable

[Alfredo Deza]

Hi,

Yesterday I completed the 14.2.4 release by jumping a few procedures
that should've been followed, mainly communicating with the release
manager and the team involved in sending out the release announcement.

There was a sense of urgency since the 14.2.3 release introduced
issues with the ceph-volume tool which breaks deployment tools that
capture stderr/stdout (see https://tracker.ceph.com/issues/41660)

The release process started on Friday (!) and finished late yesterday.
I'm following up with the release manager to get the proper notes out,
like I mentioned he was unaware of the ongoing release (my fault).

Let me know if you have any questions (or complaints!) while we try to
get this sorted out.

-Alfredo

On Tue, Sep 17, 2019 at 7:49 AM Janne Johansson  wrote:
>
>
>
> Den tis 17 sep. 2019 kl 12:52 skrev Yoann Moulin :
>>
>> Hello,
>>
>> >>> Never install packages until there is an announcement.
>> >>>
>>
>> My reaction was not on this specific release but with this sentence :  « 
>> Never install packages until there is an announcement. » And also with
>> this one : « If you need to do installs all the time, and can not postpone 
>> until the repo settle. Consider rsyncing the repo after a release,
>> and use that for installs. » This sound crazy to me.
>
>
> I would have to agree with this. If these statements are true:
> 1) devs say: don't install unless there is an announcement out
> 2) devs have previously dropped new releases before posting the announcement 
> and told people it was a mistake to "yum upgrade"/"apt upgrade" before 
> announcement
>
> then the solution seems very simple, do NOT populate the repos with packages 
> that your users shouldn't install until release notes are out.
>
> I am well aware of that 100s of things around packaging and testing a release 
> is hard, times the amount of platforms and arches you support, but to the 
> untrained eye, not pushing packages out seems like quite a binary thing. 
> Either you press whatever button there is that pushes the files out, or you 
> don't press it.
>
> If there is a reason for people to not run packages until a certain condition 
> is true, then don't press that hypothetical button until you get convinced 
> that the condition is.
>
> Among all other things that are hard about releases, that can't be the worst?
>
> --
> May the most significant bit of your life be positive.
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: upmap supported in SLES 12SPx

[Thomas Schneider]

Hi,

I was checking the connected sessions on all of my 3 MON nodes with this
command:
ceph daemon mon. sessions | grep -v luminous

This returns the following 2 clients featuring
0x27018fb86aa42ada
and
0x27018eb84aa42a52

This is mapping to OS / Kernel:
0x27018fb86aa42ada
Debian 10.1
Kernel 5.0.21-1-pve
and
SLES 12SP4
Kernel 4.12.14-95.13-default

0x27018eb84aa42a52
SLES 12SP3
4.4.176-94.88-default
and
SLES 12SP3
4.4.180-94.97-default
and
SLES 12SP4
4.4.156-94.64-default

Based on your previous information 0x27018fb86aa42ada and
0x27018eb84aa42a52 is ready for upmap.
But then I wonder why the output of ceph daemon mon. sessions  marks
these clients as "jewel"?

I have checked the installed ceph version on each client and can confirm
that it is:
ceph version 12.2 luminous

This would drive the conclusion that the ouput of ceph daemon mon.
sessions is pointing incorrectly to "jewel".

Regards
Thomas

Am 16.09.2019 um 17:36 schrieb Ilya Dryomov:
> On Mon, Sep 16, 2019 at 5:10 PM Thomas Schneider <74cmo...@gmail.com> wrote:
>> Wonderbra.
>>
>> I found some relevant sessions on 2 of 3 monitor nodes.
>> And I found some others:
>> root@ld5505:~# ceph daemon mon.ld5505 sessions | grep 0x40106b84a842a42
>> root@ld5505:~# ceph daemon mon.ld5505 sessions | grep -v luminous
>> [
>> "MonSession(client.32679861 v1:10.97.206.92:0/1183647891 is open
>> allow *, features 0x27018fb86aa42ada (jewel))",
>> "MonSession(client.32692978 v1:10.97.206.91:0/3689092992 is open
>> allow *, features 0x27018fb86aa42ada (jewel))",
>> "MonSession(client.11935413 v1:10.96.6.116:0/3187655474 is open
>> allow r, features 0x27018eb84aa42a52 (jewel))",
>> "MonSession(client.3941901 v1:10.76.179.23:0/2967896845 is open
>> allow r, features 0x27018fb86aa42ada (jewel))",
>> "MonSession(client.28313343 v1:10.76.177.108:0/1303617860 is open
>> allow r, features 0x27018fb86aa42ada (jewel))",
>> "MonSession(client.29311725 v1:10.97.206.94:0/224438037 is open
>> allow *, features 0x27018fb86aa42ada (jewel))",
>> "MonSession(client.4535833 v1:10.76.177.133:0/1269608815 is open
>> allow r, features 0x27018fb86aa42ada (jewel))",
>> "MonSession(client.3919902 v1:10.96.4.243:0/293623521 is open allow
>> r, features 0x27018eb84aa42a52 (jewel))",
>> "MonSession(client.35678944 v1:10.76.179.211:0/4218086982 is open
>> allow r, features 0x27018eb84aa42a52 (jewel))",
>> "MonSession(client.35751316 v1:10.76.179.30:0/1348696702 is open
>> allow r, features 0x27018eb84aa42a52 (jewel))",
>> "MonSession(client.28246527 v1:10.96.4.228:0/1495661381 is open
>> allow r, features 0x27018fb86aa42ada (jewel))",
>> "MonSession(client.3917843 v1:10.76.179.22:0/489863209 is open allow
>> r, features 0x27018fb86aa42ada (jewel))",
>> "MonSession(unknown.0 - is open allow r, features 0x27018eb84aa42a52
>> (jewel))",
>> ]
>>
>> Would it make sense to shutdown these clients, too?
>>
>> What confuses me is that the list includes clients that belong to the
>> Ceph cluster, namely 10.97.206.0/24.
>> All nodes of the Ceph cluster are identical in terms of OS, kernel, Ceph.
> The above output seems consistent with your "ceph features" output: it
> lists clients with features 0x27018eb84aa42a52 and 0x27018fb86aa42ada.
> Like I said in my previous email, both of these support upmap.
>
> If you temporarily shut them down, set-require-min-compat-client will
> work without --yes-i-really-mean-it.
>
> Thanks,
>
> Ilya

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Unsubscribe

[Rimma Iontel]

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Unsubscribe

[Wesley Peng]

on 2019/9/17 20:34, Rimma Iontel wrote:

To unsubscribe send an email to ceph-users-le...@ceph.io


As the signature in sent mail shows, you would send a message to 
ceph-users-le...@ceph.io for quitting.


regards.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: upmap supported in SLES 12SPx

[Eugen Block]

Hi,


I have checked the installed ceph version on each client and can confirm
that it is:
ceph version 12.2 luminous

This would drive the conclusion that the ouput of ceph daemon mon.
sessions is pointing incorrectly to "jewel".


I have the same in an upgraded Nautilus cluster. There are a couple of  
"unknown" clients and some Jewel clients, but checking those clients  
they all have at least Luminous. I rebooted some of those clients,  
too, but this didn't change anything. The min-compat-client already  
has been forced to luminous (knowing that in fact there are no jewel  
clients):


host1:~ # ceph osd get-require-min-compat-client
luminous

I'd be curious how to get rid of the jewel client sessions.

Regards,
Eugen


Zitat von Thomas Schneider <74cmo...@gmail.com>:


Hi,

I was checking the connected sessions on all of my 3 MON nodes with this
command:
ceph daemon mon. sessions | grep -v luminous

This returns the following 2 clients featuring
0x27018fb86aa42ada
and
0x27018eb84aa42a52

This is mapping to OS / Kernel:
0x27018fb86aa42ada
Debian 10.1
Kernel 5.0.21-1-pve
and
SLES 12SP4
Kernel 4.12.14-95.13-default

0x27018eb84aa42a52
SLES 12SP3
4.4.176-94.88-default
and
SLES 12SP3
4.4.180-94.97-default
and
SLES 12SP4
4.4.156-94.64-default

Based on your previous information 0x27018fb86aa42ada and
0x27018eb84aa42a52 is ready for upmap.
But then I wonder why the output of ceph daemon mon. sessions  marks
these clients as "jewel"?

I have checked the installed ceph version on each client and can confirm
that it is:
ceph version 12.2 luminous

This would drive the conclusion that the ouput of ceph daemon mon.
sessions is pointing incorrectly to "jewel".

Regards
Thomas

Am 16.09.2019 um 17:36 schrieb Ilya Dryomov:

On Mon, Sep 16, 2019 at 5:10 PM Thomas Schneider <74cmo...@gmail.com> wrote:

Wonderbra.

I found some relevant sessions on 2 of 3 monitor nodes.
And I found some others:
root@ld5505:~# ceph daemon mon.ld5505 sessions | grep 0x40106b84a842a42
root@ld5505:~# ceph daemon mon.ld5505 sessions | grep -v luminous
[
"MonSession(client.32679861 v1:10.97.206.92:0/1183647891 is open
allow *, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.32692978 v1:10.97.206.91:0/3689092992 is open
allow *, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.11935413 v1:10.96.6.116:0/3187655474 is open
allow r, features 0x27018eb84aa42a52 (jewel))",
"MonSession(client.3941901 v1:10.76.179.23:0/2967896845 is open
allow r, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.28313343 v1:10.76.177.108:0/1303617860 is open
allow r, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.29311725 v1:10.97.206.94:0/224438037 is open
allow *, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.4535833 v1:10.76.177.133:0/1269608815 is open
allow r, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.3919902 v1:10.96.4.243:0/293623521 is open allow
r, features 0x27018eb84aa42a52 (jewel))",
"MonSession(client.35678944 v1:10.76.179.211:0/4218086982 is open
allow r, features 0x27018eb84aa42a52 (jewel))",
"MonSession(client.35751316 v1:10.76.179.30:0/1348696702 is open
allow r, features 0x27018eb84aa42a52 (jewel))",
"MonSession(client.28246527 v1:10.96.4.228:0/1495661381 is open
allow r, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.3917843 v1:10.76.179.22:0/489863209 is open allow
r, features 0x27018fb86aa42ada (jewel))",
"MonSession(unknown.0 - is open allow r, features 0x27018eb84aa42a52
(jewel))",
]

Would it make sense to shutdown these clients, too?

What confuses me is that the list includes clients that belong to the
Ceph cluster, namely 10.97.206.0/24.
All nodes of the Ceph cluster are identical in terms of OS, kernel, Ceph.

The above output seems consistent with your "ceph features" output: it
lists clients with features 0x27018eb84aa42a52 and 0x27018fb86aa42ada.
Like I said in my previous email, both of these support upmap.

If you temporarily shut them down, set-require-min-compat-client will
work without --yes-i-really-mean-it.

Thanks,

Ilya


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: download.ceph.com repository changes

[Alfredo Deza]

Reviving this old thread.

I still think this is something we should consider as users still
experience problems:

* Impossible to 'pin' to a version. User installs 14.2.0 and 4 months
later they add other nodes but version moved to 14.2.2
* Impossible to use a version that is not what the latest is (e.g. if
someone doesn't need the release from Monday, but wants the one from 6
months ago), similar to the above
* When a release is underway, the repository breaks because syncing
packages takes hours. The operation is not atomic.
* It is not currently possible to "remove" a bad release, in the past,
this means cutting a new release as soon as possible, which can take
days

The latest issue (my fault!) was to cut a release and get the packages
out without communicating with the release manager, which caused users
to note there is a new version *as soon as it was up* vs, a
process that could've not touched the 'latest' url until the
announcement goes out.

If you have been affected by any of these issues (or others I didn't
come up with), please let us know in this thread so that we can find
some common ground and try to improve the process.

Thanks!

On Tue, Jul 24, 2018 at 10:38 AM Alfredo Deza  wrote:
>
> Hi all,
>
> After the 12.2.6 release went out, we've been thinking on better ways
> to remove a version from our repositories to prevent users from
> upgrading/installing a known bad release.
>
> The way our repos are structured today means every single version of
> the release is included in the repository. That is, for Luminous,
> every 12.x.x version of the binaries is in the same repo. This is true
> for both RPM and DEB repositories.
>
> However, the DEB repos don't allow pinning to a given version because
> our tooling (namely reprepro) doesn't construct the repositories in a
> way that this is allowed. For RPM repos this is fine, and version
> pinning works.
>
> To remove a bad version we have to proposals (and would like to hear
> ideas on other possibilities), one that would involve symlinks and the
> other one which purges the known bad version from our repos.
>
> *Symlinking*
> When releasing we would have a "previous" and "latest" symlink that
> would get updated as versions move forward. It would require
> separation of versions at the URL level (all versions would no longer
> be available in one repo).
>
> The URL structure would then look like:
>
> debian/luminous/12.2.3/
> debian/luminous/previous/  (points to 12.2.5)
> debian/luminous/latest/   (points to 12.2.7)
>
> Caveats: the url structure would change from debian-luminous/ to
> prevent breakage, and the versions would be split. For RPMs it would
> mean a regression if someone is used to pinning, for example pinning
> to 12.2.2 wouldn't be possible using the same url.
>
> Pros: Faster release times, less need to move packages around, and
> easier to remove a bad version
>
>
> *Single version removal*
> Our tooling would need to go and remove the known bad version from the
> repository, which would require to rebuild the repository again, so
> that the metadata is updated with the difference in the binaries.
>
> Caveats: time intensive process, almost like cutting a new release
> which takes about a day (and sometimes longer). Error prone since the
> process wouldn't be the same (one off, just when a version needs to be
> removed)
>
> Pros: all urls for download.ceph.com and its structure are kept the same.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] cephfs and selinux

[Andrey Suharev]

Hi all,

I would like to have my home dir at cephfs and to keep selinux enabled 
at the same time.


The trouble is selinux prevents sshd to access ~/.ssh/authorized_keys 
file. Any ideas how to fix it?

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: upmap supported in SLES 12SPx

[Luis Henriques]

Thomas <74cmo...@gmail.com> writes:

> But I think the number of clients older than Luminous is incorrect.
>
> I'm running 98% of the clients with SLES 12SPx, and therefore the
> question is:
> Can you confirm in which SLES 12 release the function upmap is supported?

I can confirm that SLE12-SP3 *does* support upmap; if you're running
older clients (e.g., SP2), upmap is not supported.  As Ilya already
mentioned, client 6 doesn't seem to support this feature, so I'm
assuming you're using SP2 or older there.

Cheers,
-- 
Luis


> And is it safe to execute ceph osd set-require-min-compat-client
> luminous --yes-i-really-mean-it? What happens to the clients then?
>
> Regards
> Thomas
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: dashboard not working

[Ricardo Dias]

Hi,

Try to check the mgr log for any hint on why the dashboard did not initialize 
properly.

Thanks,
Ricardo Dias


From: solarflow99 
Sent: Tuesday, September 17, 2019 00:10
To: ceph-users
Subject: [ceph-users] dashboard not working

I have mimic installed and for some reason the dashboard isn't showing up.  I 
see which mon is listed as active for "mgr", the module is enabled, but nothing 
is listening on port 8080:

# ceph mgr module ls
{
"enabled_modules": [
"dashboard",
"iostat",
"status"


tcp0  0 10.8.152.4:6800 0.0.0.0:*   
LISTEN  69049/ceph-mgr

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] (no subject)

[Amudhan P]

Hi,

I am using ceph version 13.2.6 (mimic) on test setup trying with cephfs.

My current setup:
3 nodes, 1 node contain two bricks and other 2 nodes contain single brick
each.

Volume is a 3 replica, I am trying to simulate node failure.

I powered down one host and started getting msg in other systems when
running any command
"-bash: fork: Cannot allocate memory" and system not responding to commands.

what could be the reason for this?
at this stage, I could able to read some of the data stored in the volume
and some just waiting for IO.

output from "sudo ceph -s"
  cluster:
id: 7c138e13-7b98-4309-b591-d4091a1742b4
health: HEALTH_WARN
1 osds down
2 hosts (3 osds) down
Degraded data redundancy: 5313488/7970232 objects degraded
(66.667%), 64 pgs degraded

  services:
mon: 1 daemons, quorum mon01
mgr: mon01(active)
mds: cephfs-tst-1/1/1 up  {0=mon01=up:active}
osd: 4 osds: 1 up, 2 in

  data:
pools:   2 pools, 64 pgs
objects: 2.66 M objects, 206 GiB
usage:   421 GiB used, 3.2 TiB / 3.6 TiB avail
pgs: 5313488/7970232 objects degraded (66.667%)
 64 active+undersized+degraded

  io:
client:   79 MiB/s rd, 24 op/s rd, 0 op/s wr

output from : sudo ceph osd df
ID CLASS WEIGHT  REWEIGHT SIZEUSE AVAIL   %USE  VAR  PGS
 0   hdd 1.819400 0 B 0 B 0 B 00   0
 3   hdd 1.819400 0 B 0 B 0 B 00   0
 1   hdd 1.81940  1.0 1.8 TiB 211 GiB 1.6 TiB 11.34 1.00   0
 2   hdd 1.81940  1.0 1.8 TiB 210 GiB 1.6 TiB 11.28 1.00  64
TOTAL 3.6 TiB 421 GiB 3.2 TiB 11.31
MIN/MAX VAR: 1.00/1.00  STDDEV: 0.03

regards
Amudhan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: upmap supported in SLES 12SPx

[Ilya Dryomov]

On Tue, Sep 17, 2019 at 2:54 PM Eugen Block  wrote:
>
> Hi,
>
> > I have checked the installed ceph version on each client and can confirm
> > that it is:
> > ceph version 12.2 luminous
> >
> > This would drive the conclusion that the ouput of ceph daemon mon.
> > sessions is pointing incorrectly to "jewel".
>
> I have the same in an upgraded Nautilus cluster. There are a couple of
> "unknown" clients and some Jewel clients, but checking those clients
> they all have at least Luminous. I rebooted some of those clients,
> too, but this didn't change anything. The min-compat-client already
> has been forced to luminous (knowing that in fact there are no jewel
> clients):
>
> host1:~ # ceph osd get-require-min-compat-client
> luminous
>
> I'd be curious how to get rid of the jewel client sessions.

https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/JP44CGR4FO5Y7FMCC3VYDRXNQ3T26UHY/

Thanks,

Ilya
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: download.ceph.com repository changes

[Marc Roos]

 
Just wanted to say we do not have any problems with current/past setup. 
Our ceph nodes are not even connected to the internet and we relay 
everything via 'our own local mirror'. 






-Original Message-
From: Alfredo Deza [mailto:ad...@redhat.com] 
Sent: dinsdag 17 september 2019 15:15
To: ceph-maintain...@ceph.com; ceph-users; ceph-devel
Subject: [ceph-users] Re: download.ceph.com repository changes

Reviving this old thread.

I still think this is something we should consider as users still 
experience problems:

* Impossible to 'pin' to a version. User installs 14.2.0 and 4 months 
later they add other nodes but version moved to 14.2.2
* Impossible to use a version that is not what the latest is (e.g. if 
someone doesn't need the release from Monday, but wants the one from 6 
months ago), similar to the above
* When a release is underway, the repository breaks because syncing 
packages takes hours. The operation is not atomic.
* It is not currently possible to "remove" a bad release, in the past, 
this means cutting a new release as soon as possible, which can take 
days

The latest issue (my fault!) was to cut a release and get the packages 
out without communicating with the release manager, which caused users 
to note there is a new version *as soon as it was up* vs, a process that 
could've not touched the 'latest' url until the announcement goes out.

If you have been affected by any of these issues (or others I didn't 
come up with), please let us know in this thread so that we can find 
some common ground and try to improve the process.

Thanks!

On Tue, Jul 24, 2018 at 10:38 AM Alfredo Deza  wrote:
>
> Hi all,
>
> After the 12.2.6 release went out, we've been thinking on better ways 
> to remove a version from our repositories to prevent users from 
> upgrading/installing a known bad release.
>
> The way our repos are structured today means every single version of 
> the release is included in the repository. That is, for Luminous, 
> every 12.x.x version of the binaries is in the same repo. This is true 

> for both RPM and DEB repositories.
>
> However, the DEB repos don't allow pinning to a given version because 
> our tooling (namely reprepro) doesn't construct the repositories in a 
> way that this is allowed. For RPM repos this is fine, and version 
> pinning works.
>
> To remove a bad version we have to proposals (and would like to hear 
> ideas on other possibilities), one that would involve symlinks and the 

> other one which purges the known bad version from our repos.
>
> *Symlinking*
> When releasing we would have a "previous" and "latest" symlink that 
> would get updated as versions move forward. It would require 
> separation of versions at the URL level (all versions would no longer 
> be available in one repo).
>
> The URL structure would then look like:
>
> debian/luminous/12.2.3/
> debian/luminous/previous/  (points to 12.2.5)
> debian/luminous/latest/   (points to 12.2.7)
>
> Caveats: the url structure would change from debian-luminous/ to 
> prevent breakage, and the versions would be split. For RPMs it would 
> mean a regression if someone is used to pinning, for example pinning 
> to 12.2.2 wouldn't be possible using the same url.
>
> Pros: Faster release times, less need to move packages around, and 
> easier to remove a bad version
>
>
> *Single version removal*
> Our tooling would need to go and remove the known bad version from the 

> repository, which would require to rebuild the repository again, so 
> that the metadata is updated with the difference in the binaries.
>
> Caveats: time intensive process, almost like cutting a new release 
> which takes about a day (and sometimes longer). Error prone since the 
> process wouldn't be the same (one off, just when a version needs to be
> removed)
>
> Pros: all urls for download.ceph.com and its structure are kept the 
same.
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an 
email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephfs and selinux

[Benjeman Meekhof]

You can setup a custom SELinux module to enable access.  We use the
following snippet to allow sshd to access authorized keys in home
directories on CephFS:

module local-ceph-ssh-auth 1.0;

require {
type cephfs_t;
type sshd_t;
class file { read getattr open };
}

#= sshd_t ==
allow sshd_t cephfs_t:file { read getattr open };

Compiling and persistently installing such a module is covered by
various documentation, such as:
https://wiki.centos.org/HowTos/SELinux#head-aa437f65e1c7873cddbafd9e9a73bbf9d102c072
(7.1. Manually Customizing Policy Modules).  Also covered there is
using audit2allow to create your own module from SELinux audit logs.

thanks,
Ben

On Tue, Sep 17, 2019 at 9:22 AM Andrey Suharev  wrote:
>
> Hi all,
>
> I would like to have my home dir at cephfs and to keep selinux enabled
> at the same time.
>
> The trouble is selinux prevents sshd to access ~/.ssh/authorized_keys
> file. Any ideas how to fix it?
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: download.ceph.com repository changes

[Abhishek Lekshmanan]

"Alfredo Deza"  writes:

> Reviving this old thread.
>
> I still think this is something we should consider as users still
> experience problems:
>
> * Impossible to 'pin' to a version. User installs 14.2.0 and 4 months
> later they add other nodes but version moved to 14.2.2
> * Impossible to use a version that is not what the latest is (e.g. if
> someone doesn't need the release from Monday, but wants the one from 6
> months ago), similar to the above
> * When a release is underway, the repository breaks because syncing
> packages takes hours. The operation is not atomic.

One of the main problems is the non atomicity here, so one way would be
that we announce we're building and syncing packages and the release
should be out soon, the main problem then is that this process can vary
between hours to a couple of days (or if it's a fri. longer).

> * It is not currently possible to "remove" a bad release, in the past,
> this means cutting a new release as soon as possible, which can take
> days
>
> The latest issue (my fault!) was to cut a release and get the packages
> out without communicating with the release manager, which caused users
> to note there is a new version *as soon as it was up* vs, a
> process that could've not touched the 'latest' url until the
> announcement goes out.
>
> If you have been affected by any of these issues (or others I didn't
> come up with), please let us know in this thread so that we can find
> some common ground and try to improve the process.
>
> Thanks!
>
> On Tue, Jul 24, 2018 at 10:38 AM Alfredo Deza  wrote:
>>
>> Hi all,
>>
>> After the 12.2.6 release went out, we've been thinking on better ways
>> to remove a version from our repositories to prevent users from
>> upgrading/installing a known bad release.
>>
>> The way our repos are structured today means every single version of
>> the release is included in the repository. That is, for Luminous,
>> every 12.x.x version of the binaries is in the same repo. This is true
>> for both RPM and DEB repositories.
>>
>> However, the DEB repos don't allow pinning to a given version because
>> our tooling (namely reprepro) doesn't construct the repositories in a
>> way that this is allowed. For RPM repos this is fine, and version
>> pinning works.
>>
>> To remove a bad version we have to proposals (and would like to hear
>> ideas on other possibilities), one that would involve symlinks and the
>> other one which purges the known bad version from our repos.
>>
>> *Symlinking*
>> When releasing we would have a "previous" and "latest" symlink that
>> would get updated as versions move forward. It would require
>> separation of versions at the URL level (all versions would no longer
>> be available in one repo).
>>
>> The URL structure would then look like:
>>
>> debian/luminous/12.2.3/
>> debian/luminous/previous/  (points to 12.2.5)
>> debian/luminous/latest/   (points to 12.2.7)
>>
>> Caveats: the url structure would change from debian-luminous/ to
>> prevent breakage, and the versions would be split. For RPMs it would
>> mean a regression if someone is used to pinning, for example pinning
>> to 12.2.2 wouldn't be possible using the same url.
>>
>> Pros: Faster release times, less need to move packages around, and
>> easier to remove a bad version
>>
>>
>> *Single version removal*
>> Our tooling would need to go and remove the known bad version from the
>> repository, which would require to rebuild the repository again, so
>> that the metadata is updated with the difference in the binaries.
>>
>> Caveats: time intensive process, almost like cutting a new release
>> which takes about a day (and sometimes longer). Error prone since the
>> process wouldn't be the same (one off, just when a version needs to be
>> removed)
>>
>> Pros: all urls for download.ceph.com and its structure are kept the same.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>

-- 
Abhishek 
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: download.ceph.com repository changes

[James Dingwall]

On 17/09/2019 14:14, Alfredo Deza wrote:
> * Impossible to 'pin' to a version. User installs 14.2.0 and 4 months
> later they add other nodes but version moved to 14.2.2

I dynamically generate a pin for the ceph .deb files in ansible using
the tasks below.  IIRC the ceph-deploy package doesn't follow the same
versioning but I'm not using that tool.  Hope others might find this useful.

James

- name: "get ceph Package list"
   set_fact:
 ceph_version: "14.2.3*"
 ceph_packages: "{{ lookup('url',
'http://download.ceph.com/debian-nautilus/dists/bionic/main/binary-amd64/Packages',
wantlist=True) | select('match', '^Package: *') | sort | list }}"
   run_once: yes

- name: "generate apt pin for ceph"
   template:
 src: "{{ playbook_dir }}/tasks/apt-ceph-pin.j2"
 dest: /etc/apt/preferences.d/ceph
 mode: 0644
 owner: root
 group: root


The template is:

{% for pin in ceph_packages %}
{{ pin }}
Pin: version {{ ceph_version }}
Pin-Priority: 1001

{% endfor %}
Zynstra is a private limited company registered in England and Wales 
(registered number 07864369). Our registered office and Headquarters are at The 
Innovation Centre, Broad Quay, Bath, BA1 1UD. This email, its contents and any 
attachments are confidential. If you have received this message in error please 
delete it from your system and advise the sender immediately.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: download.ceph.com repository changes

[Janne Johansson]

Den tis 17 sep. 2019 kl 15:15 skrev Alfredo Deza :

> Reviving this old thread.
> * When a release is underway, the repository breaks because syncing
> packages takes hours. The operation is not atomic.
>

Couldn't they be almost atomic?
I believe both "yum" and "apt" would only consider rpms/debs if they are
listed in the repo index files*, so you should be able to copy out rpms and
debs and at the final step "mv" the index files into place, making the
packages visible in a super short time?

*) .xml.gz for rpms, db/ files for debs.

Obviously this is a very small part of the list of problems, but I think
anyone mirroring huge sets of rpms/debs would have faced and solved this
particular nit by now?


-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: download.ceph.com repository changes

[vitalif]

The worst part about the official repository is that it lacks Debian 
packages


Also of course it would be very convenient to be able to install any 
version from the repos, not just the latest one. It's certainly possible 
with debian repos...

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: verify_upmap number of buckets 5 exceeds desired 4

[Eric Dold]

With ceph 14.2.4 it's the same.
The upmap balancer is not working.

Any ideas?

On Wed, Sep 11, 2019 at 11:32 AM Eric Dold  wrote:

> Hello,
>
> I'm running ceph 14.2.3 on six hosts with each four osds. I did recently
> upgrade this from four hosts.
>
> The cluster is running fine. But i get this in my logs:
>
> Sep 11 11:02:41 ceph1 ceph-mon[1333]: 2019-09-11 11:02:41.953 7f26023a6700
> -1 verify_upmap number of buckets 5 exceeds desired 4
> Sep 11 11:02:41 ceph1 ceph-mon[1333]: 2019-09-11 11:02:41.953 7f26023a6700
> -1 verify_upmap number of buckets 5 exceeds desired 4
> Sep 11 11:02:41 ceph1 ceph-mon[1333]: 2019-09-11 11:02:41.953 7f26023a6700
> -1 verify_upmap number of buckets 5 exceeds desired 4
>
> It looks like the balancer is not doing any work.
>
> Here are some infos about the cluster:
>
> ceph1 ~ # ceph osd crush rule ls
> replicated_rule
> cephfs_ec
> ceph1 ~ # ceph osd crush rule dump replicated_rule
> {
> "rule_id": 0,
> "rule_name": "replicated_rule",
> "ruleset": 0,
> "type": 1,
> "min_size": 1,
> "max_size": 10,
> "steps": [
> {
> "op": "take",
> "item": -1,
> "item_name": "default"
> },
> {
> "op": "chooseleaf_firstn",
> "num": 0,
> "type": "host"
> },
> {
> "op": "emit"
> }
> ]
> }
>
> ceph1 ~ # ceph osd crush rule dump cephfs_ec
> {
> "rule_id": 1,
> "rule_name": "cephfs_ec",
> "ruleset": 1,
> "type": 3,
> "min_size": 8,
> "max_size": 8,
> "steps": [
> {
> "op": "set_chooseleaf_tries",
> "num": 5
> },
> {
> "op": "set_choose_tries",
> "num": 100
> },
> {
> "op": "take",
> "item": -1,
> "item_name": "default"
> },
> {
> "op": "choose_indep",
> "num": 4,
> "type": "host"
> },
> {
> "op": "choose_indep",
> "num": 2,
> "type": "osd"
> },
> {
> "op": "emit"
> }
> ]
> }
>
> ceph1 ~ # ceph osd erasure-code-profile ls
> default
> isa_62
> ceph1 ~ # ceph osd erasure-code-profile get default
> k=2
> m=1
> plugin=jerasure
> technique=reed_sol_van
> ceph1 ~ # ceph osd erasure-code-profile get isa_62
> crush-device-class=
> crush-failure-domain=osd
> crush-root=default
> k=6
> m=2
> plugin=isa
> technique=reed_sol_van
>
> The idea with four hosts was that the ec profile should take two osds on
> each host for the eight buckets.
> Now with six hosts i guess two hosts will have tow buckets on two osds and
> four hosts will have each one bucket for a piece of data.
>
> Any idea how to resolve this?
>
> Regards
> Eric
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: download.ceph.com repository changes

[Sasha Litvak]

I have been affected by few issues mentioned by Alfredo.

* Version Pinning:  Had to install several debs of specific version to be
able to pull dependencies of the correct version.  I believe that other
projects resolving it by creating a virtual package that pulls all of the
proper dependencies in.  Not sure if the same done by RPM / Yum.

* Unannounced releases.  I believe it is more of a procedural issue and
unfortunately something will need to be done to enforce the compliance once
rules of packages release are finalized.

* I am bothered with a quality of the releases of a very complex system
that can bring down a whole house and keep it down for a while.  While I
wish the QA would be perfect, I wonder if it would be practical to release
new packages to a testing repo before moving it to a main one.  There is a
chance then someone will detect a problem before it becomes a
production issue.   Let it seat for a couple days or weeks in testing.
People who need new update right away or just want to test will install it
and report the problems.  Others will not be affected.

Just my 2c,

On Tue, Sep 17, 2019, 8:15 AM Alfredo Deza  wrote:

> Reviving this old thread.
>
> I still think this is something we should consider as users still
> experience problems:
>
> * Impossible to 'pin' to a version. User installs 14.2.0 and 4 months
> later they add other nodes but version moved to 14.2.2
> * Impossible to use a version that is not what the latest is (e.g. if
> someone doesn't need the release from Monday, but wants the one from 6
> months ago), similar to the above
> * When a release is underway, the repository breaks because syncing
> packages takes hours. The operation is not atomic.
> * It is not currently possible to "remove" a bad release, in the past,
> this means cutting a new release as soon as possible, which can take
> days
>
> The latest issue (my fault!) was to cut a release and get the packages
> out without communicating with the release manager, which caused users
> to note there is a new version *as soon as it was up* vs, a
> process that could've not touched the 'latest' url until the
> announcement goes out.
>
> If you have been affected by any of these issues (or others I didn't
> come up with), please let us know in this thread so that we can find
> some common ground and try to improve the process.
>
> Thanks!
>
> On Tue, Jul 24, 2018 at 10:38 AM Alfredo Deza  wrote:
> >
> > Hi all,
> >
> > After the 12.2.6 release went out, we've been thinking on better ways
> > to remove a version from our repositories to prevent users from
> > upgrading/installing a known bad release.
> >
> > The way our repos are structured today means every single version of
> > the release is included in the repository. That is, for Luminous,
> > every 12.x.x version of the binaries is in the same repo. This is true
> > for both RPM and DEB repositories.
> >
> > However, the DEB repos don't allow pinning to a given version because
> > our tooling (namely reprepro) doesn't construct the repositories in a
> > way that this is allowed. For RPM repos this is fine, and version
> > pinning works.
> >
> > To remove a bad version we have to proposals (and would like to hear
> > ideas on other possibilities), one that would involve symlinks and the
> > other one which purges the known bad version from our repos.
> >
> > *Symlinking*
> > When releasing we would have a "previous" and "latest" symlink that
> > would get updated as versions move forward. It would require
> > separation of versions at the URL level (all versions would no longer
> > be available in one repo).
> >
> > The URL structure would then look like:
> >
> > debian/luminous/12.2.3/
> > debian/luminous/previous/  (points to 12.2.5)
> > debian/luminous/latest/   (points to 12.2.7)
> >
> > Caveats: the url structure would change from debian-luminous/ to
> > prevent breakage, and the versions would be split. For RPMs it would
> > mean a regression if someone is used to pinning, for example pinning
> > to 12.2.2 wouldn't be possible using the same url.
> >
> > Pros: Faster release times, less need to move packages around, and
> > easier to remove a bad version
> >
> >
> > *Single version removal*
> > Our tooling would need to go and remove the known bad version from the
> > repository, which would require to rebuild the repository again, so
> > that the metadata is updated with the difference in the binaries.
> >
> > Caveats: time intensive process, almost like cutting a new release
> > which takes about a day (and sometimes longer). Error prone since the
> > process wouldn't be the same (one off, just when a version needs to be
> > removed)
> >
> > Pros: all urls for download.ceph.com and its structure are kept the
> same.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing lis

[ceph-users] Re: upmap supported in SLES 12SPx

[Eugen Block]

Thank you very much, Ilya!


Zitat von Ilya Dryomov :


On Tue, Sep 17, 2019 at 2:54 PM Eugen Block  wrote:


Hi,

> I have checked the installed ceph version on each client and can confirm
> that it is:
> ceph version 12.2 luminous
>
> This would drive the conclusion that the ouput of ceph daemon mon.
> sessions is pointing incorrectly to "jewel".

I have the same in an upgraded Nautilus cluster. There are a couple of
"unknown" clients and some Jewel clients, but checking those clients
they all have at least Luminous. I rebooted some of those clients,
too, but this didn't change anything. The min-compat-client already
has been forced to luminous (knowing that in fact there are no jewel
clients):

host1:~ # ceph osd get-require-min-compat-client
luminous

I'd be curious how to get rid of the jewel client sessions.


https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/JP44CGR4FO5Y7FMCC3VYDRXNQ3T26UHY/

Thanks,

Ilya



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: osds xxx have blocked requests > 1048.58 sec / osd.yyy has stuck requests > 67108.9 sec

[Robert LeBlanc]

Not sure which version you are on, but adding these to your
/etc/ceph/ceph.conf file and restarting the OSD processes can go a long way
to helping these really long blocks. It won't get rid of them completely,
but should really help.

osd op queue = wpq
osd op queue cut off = high


Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Mon, Sep 16, 2019 at 11:34 PM Thomas <74cmo...@gmail.com> wrote:

> Hi,
>
> I have defined pool hdd which is exclusively used by virtual disks of
> multiple KVMs / LXCs.
> Yesterday I run these commands
> osdmaptool om --upmap out.txt --upmap-pool hdd
> source out.txt
> and Ceph started rebalancing this pool.
>
> However since then no KVM / LXC is reacting anymore.
> If I try to start a new KVM it hangs in boot process.
>
> This is the output of ceph health detail:
> root@ld3955:/mnt/rbd# ceph health detail
> HEALTH_ERR 28 nearfull osd(s); 1 pool(s) nearfull; Reduced data
> availability: 1 pg inactive, 1 pg peering; Degraded data redundancy (low
> space): 8 pgs backfill_toofull; 1 subtrees have overcommitted pool
> target_size_bytes; 1 subtrees have overcommitted pool target_size_ratio;
> 2 pools have too many placement groups; 672 slow requests are blocked >
> 32 sec; 4752 stuck requests are blocked > 4096 sec
> OSD_NEARFULL 28 nearfull osd(s)
> osd.42 is near full
> osd.44 is near full
> osd.45 is near full
> osd.77 is near full
> osd.84 is near full
> osd.94 is near full
> osd.101 is near full
> osd.103 is near full
> osd.106 is near full
> osd.109 is near full
> osd.113 is near full
> osd.118 is near full
> osd.120 is near full
> osd.136 is near full
> osd.138 is near full
> osd.142 is near full
> osd.147 is near full
> osd.156 is near full
> osd.159 is near full
> osd.161 is near full
> osd.168 is near full
> osd.192 is near full
> osd.202 is near full
> osd.206 is near full
> osd.208 is near full
> osd.226 is near full
> osd.234 is near full
> osd.247 is near full
> POOL_NEARFULL 1 pool(s) nearfull
> pool 'hdb_backup' is nearfull
> PG_AVAILABILITY Reduced data availability: 1 pg inactive, 1 pg peering
> pg 30.1b9 is stuck peering for 4722.750977, current state peering,
> last acting [183,27,63]
> PG_DEGRADED_FULL Degraded data redundancy (low space): 8 pgs
> backfill_toofull
> pg 11.465 is active+remapped+backfill_wait+backfill_toofull, acting
> [308,351,58]
> pg 11.5c4 is active+remapped+backfill_wait+backfill_toofull, acting
> [318,336,54]
> pg 11.afd is active+remapped+backfill_wait+backfill_toofull, acting
> [347,220,315]
> pg 11.b82 is active+remapped+backfill_toofull, acting [314,320,254]
> pg 11.1803 is active+remapped+backfill_wait+backfill_toofull, acting
> [88,363,302]
> pg 11.1aac is active+remapped+backfill_wait+backfill_toofull, acting
> [328,275,95]
> pg 11.1c09 is active+remapped+backfill_wait+backfill_toofull, acting
> [55,124,278]
> pg 11.1e36 is active+remapped+backfill_wait+backfill_toofull, acting
> [351,92,315]
> POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted pool
> target_size_bytes
> Pools ['hdb_backup'] overcommit available storage by 1.708x due to
> target_size_bytes0  on pools []
> POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted pool
> target_size_ratio
> Pools ['hdb_backup'] overcommit available storage by 1.708x due to
> target_size_ratio 0.000 on pools []
> POOL_TOO_MANY_PGS 2 pools have too many placement groups
> Pool hdd has 512 placement groups, should have 128
> Pool pve_cephfs_metadata has 32 placement groups, should have 4
> REQUEST_SLOW 672 slow requests are blocked > 32 sec
> 249 ops are blocked > 2097.15 sec
> 284 ops are blocked > 1048.58 sec
> 108 ops are blocked > 524.288 sec
> 9 ops are blocked > 262.144 sec
> 22 ops are blocked > 131.072 sec
> osd.9 has blocked requests > 524.288 sec
> osds 0,2,6,68 have blocked requests > 1048.58 sec
> osd.3 has blocked requests > 2097.15 sec
> REQUEST_STUCK 4752 stuck requests are blocked > 4096 sec
> 1431 ops are blocked > 67108.9 sec
> 513 ops are blocked > 33554.4 sec
> 909 ops are blocked > 16777.2 sec
> 1809 ops are blocked > 8388.61 sec
> 90 ops are blocked > 4194.3 sec
> osd.63 has stuck requests > 67108.9 sec
>
>
> My interpretation is that Ceph
> a) is busy with remapping PGs of pool hdb_backup
> b) has identified several OSDs with either blocked or stuck requests.
>
> Any of these OSDs belongs to pool hdd, though.
> osd.9 belongs to node A, osd.63 and osd.68 belongs to node C (there are
> 4 nodes serving OSD in the cluster).
>
> I have tried to fix this issue, but it failed with
> - ceph osd set noout
> - restart of relevant OSD by systemctl restart ceph-osd@
> and finally server reboot.
>
> I also tried to migrate the virtual disks to another pool, but this
> fails, too

[ceph-users] v14.2.4 Nautilus released

[Abhishek Lekshmanan]

This is the fourth release in the Ceph Nautilus stable release series. Its sole
purpose is to fix a regression that found its way into the previous release.

Notable Changes
---
The ceph-volume in Nautilus v14.2.3 was found to contain a serious
regression, described in https://tracker.ceph.com/issues/41660, which
prevented deployment tools like ceph-ansible, DeepSea, Rook, etc. from
deploying/removing OSDs.

Getting Ceph


* Git at git://github.com/ceph/ceph.git
* Tarball at http://download.ceph.com/tarballs/ceph-14.2.3.tar.gz
* For packages, see http://docs.ceph.com/docs/master/install/get-packages/
* Release git sha1: 75f4de193b3ea58512f204623e6c5a16e6c1e1ba

--
Abhishek Lekshmanan
SUSE Software Solutions Germany GmbH
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: v14.2.4 Nautilus released

[Abhishek Lekshmanan]

Abhishek Lekshmanan  writes:

> This is the fourth release in the Ceph Nautilus stable release series. Its 
> sole
> purpose is to fix a regression that found its way into the previous release.
>
> Notable Changes
> ---
> The ceph-volume in Nautilus v14.2.3 was found to contain a serious
> regression, described in https://tracker.ceph.com/issues/41660, which
> prevented deployment tools like ceph-ansible, DeepSea, Rook, etc. from
> deploying/removing OSDs.
>
> Getting Ceph
> 
>
> * Git at git://github.com/ceph/ceph.git
> * Tarball at http://download.ceph.com/tarballs/ceph-14.2.3.tar.gz
http://download.ceph.com/tarballs/ceph-14.2.4.tar.gz is the correct tarball

> * For packages, see http://docs.ceph.com/docs/master/install/get-packages/
> * Release git sha1: 75f4de193b3ea58512f204623e6c5a16e6c1e1ba
>
> --
> Abhishek Lekshmanan
> SUSE Software Solutions Germany GmbH
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Ceph deployment tool suggestions

[Shain Miley]

Hello,

We currently have an existing 20 node Ceph cluster (17 osd nodes with 3 
mon nodes).  When this was originally configured, much of the OS install 
was done manually and the cluster was mainly deployed using ceph-deploy.


We are going to be replacing 9 of the 1st gen nodes with 6 newer more 
dense nodes in the near future.


This time around I would like to see about automating the process as 
much as possible (both OS and Ceph installs).


I was wondering if anyone had any suggestions about the best tool to use 
for this as we are not setting up a cluster from scratch and whatever 
tool we decide to use, would have to work within the context of an 
already in production cluster.


Some of these tools seem best suited to work when you are setting up a 
cluster for the very first time, however we are not in that position at 
this point and I want to make sure I am using something that is flexible 
enough for our environment going forward.


Thanks in advance,

Shain

--
NPR | Shain Miley | Manager of Infrastructure, Digital Media | smi...@npr.org | 
202.513.3649
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] rados.py with Python3

[tomascribb]

Hi, i'm integrating ceph like glance's backend, my Openstack is stein over 
Ubuntu 18.04. When I try upload an image rados.py have an error when calls 
"run_in_thread()"  function (line 238). I've tried use rados mudule from 
python3 interface  and i have the same issue.
I read that Python 3 has different behaviour of ctypes c_char_p. 
How can I resolve that issue??
I copy that example:


admin@control1:~$ sudo python3
[sudo] password for admin:
Python 3.6.8 (default, Aug 20 2019, 17:12:48)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import rados
>>> cluster = rados.Rados(conffile='')
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/local/lib/python3.6/dist-packages/rados.py", line 239, in __init__
(byref(self.cluster), c_char_p(clustername)),
TypeError: bytes or integer address expected instead of str instance
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph deployment tool suggestions

[Paul Emmerich]

The best tool to automate both OS and Ceph deployment is ours: https://croit.io/

Check out our demo: https://croit.io/croit-virtual-demo

Paul

-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90

On Tue, Sep 17, 2019 at 4:45 PM Shain Miley  wrote:
>
> Hello,
>
> We currently have an existing 20 node Ceph cluster (17 osd nodes with 3
> mon nodes).  When this was originally configured, much of the OS install
> was done manually and the cluster was mainly deployed using ceph-deploy.
>
> We are going to be replacing 9 of the 1st gen nodes with 6 newer more
> dense nodes in the near future.
>
> This time around I would like to see about automating the process as
> much as possible (both OS and Ceph installs).
>
> I was wondering if anyone had any suggestions about the best tool to use
> for this as we are not setting up a cluster from scratch and whatever
> tool we decide to use, would have to work within the context of an
> already in production cluster.
>
> Some of these tools seem best suited to work when you are setting up a
> cluster for the very first time, however we are not in that position at
> this point and I want to make sure I am using something that is flexible
> enough for our environment going forward.
>
> Thanks in advance,
>
> Shain
>
> --
> NPR | Shain Miley | Manager of Infrastructure, Digital Media | smi...@npr.org 
> | 202.513.3649
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: dashboard not working

[solarflow99]

ok, the hint to me was cephmgr services didn't show a URL, just the module
names.  This last step was all it took:

Since each ceph-mgr hosts its own instance of dashboard, it may also be
necessary to configure them separately. The IP address and port for a
specific manager instance can be changed with the following commands:

$ ceph config set mgr mgr/dashboard/$name/server_addr $IP
$ ceph config set mgr mgr/dashboard/$name/server_port $PORT


https://docs.ceph.com/docs/mimic/mgr/dashboard/



On Tue, Sep 17, 2019 at 1:59 AM Lenz Grimmer  wrote:

> On 9/17/19 9:21 AM, solarflow99 wrote:
>
> > I'm not trying to use SSL.  It seems strange how the mgr log doesn't
> > even mention anything about dashboard
>
> Did you try running "ceph mgr services"? Does it show the dashboard as
> running? Did you check the Mgr of the current active Mgr?
>
> Lenz
>
> --
> SUSE Software Solutions Germany GmbH - Maxfeldstr. 5 - 90409 Nuernberg
> GF: Felix Imendörffer, HRB 247165 (AG Nürnberg)
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph deployment tool suggestions

[solarflow99]

can you just do a kickstart and use ceph-ansible?



On Tue, Sep 17, 2019 at 9:59 AM Paul Emmerich 
wrote:

> The best tool to automate both OS and Ceph deployment is ours:
> https://croit.io/
>
> Check out our demo: https://croit.io/croit-virtual-demo
>
> Paul
>
> --
> Paul Emmerich
>
> Looking for help with your Ceph cluster? Contact us at https://croit.io
>
> croit GmbH
> Freseniusstr. 31h
> 81247 München
> www.croit.io
> Tel: +49 89 1896585 90
>
> On Tue, Sep 17, 2019 at 4:45 PM Shain Miley  wrote:
> >
> > Hello,
> >
> > We currently have an existing 20 node Ceph cluster (17 osd nodes with 3
> > mon nodes).  When this was originally configured, much of the OS install
> > was done manually and the cluster was mainly deployed using ceph-deploy.
> >
> > We are going to be replacing 9 of the 1st gen nodes with 6 newer more
> > dense nodes in the near future.
> >
> > This time around I would like to see about automating the process as
> > much as possible (both OS and Ceph installs).
> >
> > I was wondering if anyone had any suggestions about the best tool to use
> > for this as we are not setting up a cluster from scratch and whatever
> > tool we decide to use, would have to work within the context of an
> > already in production cluster.
> >
> > Some of these tools seem best suited to work when you are setting up a
> > cluster for the very first time, however we are not in that position at
> > this point and I want to make sure I am using something that is flexible
> > enough for our environment going forward.
> >
> > Thanks in advance,
> >
> > Shain
> >
> > --
> > NPR | Shain Miley | Manager of Infrastructure, Digital Media |
> smi...@npr.org | 202.513.3649
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph deployment tool suggestions

[Benjeman Meekhof]

We have had success using Foreman to deploy OS in various roles across
multiple sites.  It can also spin up VM on various infrastructure as
part of that process.   Foreman essentially generates whatever type of
'kickstart' your OS requires on the fly and boots nodes via the
network.  It's a fairly mature tool which provides an installer that
works well to get started.  There is an Ansible plugin for Foreman but
I have no experience with it and Foreman is historically geared for
integration with Puppet.   We use Puppet to manage initial and ongoing
configuration including Ceph with the module linked below (we use a
forked version).  There's no requirement that all of your cluster
nodes be managed with the module as long as you provide it with the
right information to bootstrap components into your current cluster:
https://github.com/openstack/puppet-ceph

thanks,
Ben


On Tue, Sep 17, 2019 at 3:28 PM solarflow99  wrote:
>
> can you just do a kickstart and use ceph-ansible?
>
>
>
> On Tue, Sep 17, 2019 at 9:59 AM Paul Emmerich  wrote:
>>
>> The best tool to automate both OS and Ceph deployment is ours: 
>> https://croit.io/
>>
>> Check out our demo: https://croit.io/croit-virtual-demo
>>
>> Paul
>>
>> --
>> Paul Emmerich
>>
>> Looking for help with your Ceph cluster? Contact us at https://croit.io
>>
>> croit GmbH
>> Freseniusstr. 31h
>> 81247 München
>> www.croit.io
>> Tel: +49 89 1896585 90
>>
>> On Tue, Sep 17, 2019 at 4:45 PM Shain Miley  wrote:
>> >
>> > Hello,
>> >
>> > We currently have an existing 20 node Ceph cluster (17 osd nodes with 3
>> > mon nodes).  When this was originally configured, much of the OS install
>> > was done manually and the cluster was mainly deployed using ceph-deploy.
>> >
>> > We are going to be replacing 9 of the 1st gen nodes with 6 newer more
>> > dense nodes in the near future.
>> >
>> > This time around I would like to see about automating the process as
>> > much as possible (both OS and Ceph installs).
>> >
>> > I was wondering if anyone had any suggestions about the best tool to use
>> > for this as we are not setting up a cluster from scratch and whatever
>> > tool we decide to use, would have to work within the context of an
>> > already in production cluster.
>> >
>> > Some of these tools seem best suited to work when you are setting up a
>> > cluster for the very first time, however we are not in that position at
>> > this point and I want to make sure I am using something that is flexible
>> > enough for our environment going forward.
>> >
>> > Thanks in advance,
>> >
>> > Shain
>> >
>> > --
>> > NPR | Shain Miley | Manager of Infrastructure, Digital Media | 
>> > smi...@npr.org | 202.513.3649
>> > ___
>> > ceph-users mailing list -- ceph-users@ceph.io
>> > To unsubscribe send an email to ceph-users-le...@ceph.io
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Stability of cephfs snapshot in nautilus

[pinepinebrook]

Is snapshot of cephfs in version nautilus is production ready?
When I take a snapshot, what will happen if somebody change the content in that 
directory before the snapshot finish.
Will there be any conflict or something to notice about when I take a snapshot 
on a busy directory (probably 20 clients r/w in in that directory)
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io