[ceph-users] Re: Ceph is constantly scrubbing 1/4 of all PGs and still have pigs not scrubbed in time

[Eugen Block]

Are the scrubs eventually reported as "scrub ok" in the OSD logs? How  
long do the scrubs take? Do you see updated timestamps in the 'ceph pg  
dump' output (column DEEP_SCRUB_STAMP)?


Zitat von thymus_03fumb...@icloud.com:

I recently switched from 16.2.x to 18.2.x and migrated to cephadm,  
since the switch the cluster is constantly scrubbing, 24/7 up to 50  
PGs simultaneously and up to 20 deep scrubs simultaneously in a  
cluster that has only 12 (in use) OSDs.
Furthermore it still manages to regularly have a warning with ‘pgs  
not scrubbed in time’


I have tried various settings, like osd_deep_scrub_interval,  
osd_max_scrubs, mds_max_scrub_ops_in_progress etc.

All those get ignored.

Please advice.

Here is an output of ceos config dump:

WHO MASK  LEVEL OPTION
VALUE 
  RO
globaladvanced  auth_client_required  
cephx 
  *
globaladvanced  auth_cluster_required 
cephx 
  *
globaladvanced  auth_service_required 
cephx 
  *
globaladvanced  auth_supported
cephx 
  *
globalbasic container_image   
 
quay.io/ceph/ceph@sha256:aca35483144ab3548a7f670db9b79772e6fc51167246421c66c0bd56a6585468   
*
globalbasic device_failure_prediction_mode
local

globaladvanced  mon_allow_pool_deletetrue
globaladvanced  mon_data_avail_warn  20
globaladvanced  mon_max_pg_per_osd   400
globaladvanced  osd_max_pg_per_osd_hard_ratio 
10.00

globaladvanced  osd_pool_default_pg_autoscale_mode   on
mon   advanced  auth_allow_insecure_global_id_reclaim 
false
mon   advanced  mon_crush_min_required_version
firefly   
  *
mon   advanced  mon_warn_on_pool_no_redundancy
false
mon   advanced  public_network
10.79.0.0/16  
  *

mgr   advanced  mgr/balancer/active  true
mgr   advanced  mgr/balancer/mode 
upmap
mgr   advanced   
mgr/cephadm/manage_etc_ceph_ceph_conf_hosts  label:admin  
   *
mgr   advanced  mgr/cephadm/migration_current 
6 
  *
mgr   advanced  mgr/dashboard/GRAFANA_API_PASSWORD
admin 
  *
mgr   advanced  mgr/dashboard/GRAFANA_API_SSL_VERIFY  
false 
  *
mgr   advanced  mgr/dashboard/GRAFANA_API_URL 
https://10.79.79.12:3000  
  *
mgr   advanced  mgr/dashboard/PROMETHEUS_API_HOST 
http://10.79.79.12:9095   
  *

mgr   advanced  mgr/devicehealth/enable_monitoring   true
mgr   advanced  mgr/orchestrator/orchestrator 
cephadm

osd   advanced  osd_map_cache_size   250
osd   advanced  osd_map_share_max_epochs 50
osd   advanced  osd_mclock_profile
high_client_ops

osd   advanced  osd_pg_epoch_persisted_max_stale 50
osd.0 basic osd_mclock_max_capacity_iops_hdd  
380.869888
osd.1 basic osd_mclock_max_capacity_iops_hdd  
441.00
osd.10basic osd_mclock_max_capacity_iops_ssd  
13677.906485
osd.11basic osd_mclock_max_capacity_iops_hdd  
274.411212
osd.13basic osd_mclock_max_capacity_iops_hdd  
198.492501
osd.2 basic osd_mclock_max_capacity_iops_hdd  
251.592009
osd.3 basic osd_mclock_max_capacity_iops_

[ceph-users] Re: Unable to map RBDs after running pg-upmap-primary on the pool

[Torkil Svensgaard]

On 07/03/2024 08:52, Torkil Svensgaard wrote:

Hi

I tried to do offline read optimization[1] this morning but I am now 
unable to map the RBDs in the pool.


I did this prior to running the pg-upmap-primary commands suggested by 
the optimizer, as suggested by the latest documentation[2]:


"
ceph osd set-require-min-compat-client reef
"

The command didn't complain and the documentation stated it should 
failed if any pre-reef clients were connected so I thought all was well 
and rad the pg_upmap_primary commands.


Can I simply remove the applied pg_upmap_primary settings somehow to my 
RBDs back while investigating?


Found the rm-pg-upmap-primary command. Removing all upmaps made it 
possible to map the RBD again.


Still not sure why this didn't work though.

"
[ceph: root@lazy /]# ceph osd dump|grep require_min_compat_client
require_min_compat_client reef
"

That's seems to have stuck but we most certainly have pacific clients so 
according to the documentation the "ceph osd 
set-require-min-compat-client reef" command should have failed.


Then there's this:

"
[ceph: root@lazy /]# ceph features
{
"mon": [
{
"features": "0x3f01cfbd",
"release": "luminous",
"num": 5
}
],
"mds": [
{
"features": "0x3f01cfbd",
"release": "luminous",
"num": 3
}
],
"osd": [
{
"features": "0x3f01cfbd",
"release": "luminous",
"num": 452
}
],
"client": [
{
"features": "0x2f018fb87aa4aafe",
"release": "luminous",
"num": 13
},
{
"features": "0x3f01cfb8ffec",
"release": "luminous",
"num": 19
},
{
"features": "0x3f01cfb9fffd",
"release": "luminous",
"num": 17
},
{
"features": "0x3f01cfbf7ffd",
"release": "luminous",
"num": 14
},
{
"features": "0x3f01cfbd",
"release": "luminous",
"num": 17
}
],
"mgr": [
{
"features": "0x3f01cfbd",
"release": "luminous",
"num": 3
}
]
}
"

This suggests that min compat client reef might be set but is not in effect?

And lastly it's one thing not being able to map pg-upmap-primary RBDs 
from a pre reef client but it didn't work with a reef client either.


Mvh.

Torkil



Mvh.

Torkil

[1] https://docs.ceph.com/en/reef/rados/operations/read-balancer/
[2] https://docs.ceph.com/en/latest/rados/operations/read-balancer/


--
Torkil Svensgaard
Sysadmin
MR-Forskningssektionen, afs. 714
DRCMR, Danish Research Centre for Magnetic Resonance
Hvidovre Hospital
Kettegård Allé 30
DK-2650 Hvidovre
Denmark
Tel: +45 386 22828
E-mail: tor...@drcmr.dk
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Running dedicated RGWs for async tasks

[Marc Singer]

Hello Ceph Users

Since we are running a big S3 cluster we would like to externalize the 
RGW daemons that do async tasks, like:


 * Garbage collection
 * Lifecycle policies
 * Calculating and updating quotas

Would this be possible to do in the configuration? Which config values 
would I need to set on the exposed RGWs and which on the Async Task RGWs?


Thanks for your input.

Marc
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Remove cluster_network without routing

[Torkil Svensgaard]

On 13/02/2024 13:31, Torkil Svensgaard wrote:

Hi

Cephadm Reef 18.2.0.

We would like to remove our cluster_network without stopping the cluster 
and without having to route between the networks.


global    advanced  cluster_network    192.168.100.0/24 
   *
global    advanced  public_network    172.21.12.0/22 
   *


The documentation[1] states:

"
You may specifically assign static IP addresses or override 
cluster_network settings using the cluster_addr setting for specific OSD 
daemons.

"

So for one OSD at a time I could set cluster_addr to override the 
cluster_network IP and use the public_network IP instead? As the 
containers are using host networking they have access to both IPs and 
will just layer 2 the traffic, avoiding routing?


When all OSDs are running with a public_network IP set via cluster_addr 
we can just delete the cluster_network setting and then remove all the 
ceph_addr settings, as with no cluster_network setting the 
public_network setting will be used?


We tried with one OSD and it seems to work. Anyone see a problem with 
this approach?


Turned out to be even simpler for this setup where the OSD containers 
have access til both host networks:


1) ceph config rm global cluster_network -> nothing happened, no 
automatic redeploy or restart


2) Restart OSDs

Mvh.

Torkil


Thanks

Torkil

[1] 
https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#id3





--
Torkil Svensgaard
Sysadmin
MR-Forskningssektionen, afs. 714
DRCMR, Danish Research Centre for Magnetic Resonance
Hvidovre Hospital
Kettegård Allé 30
DK-2650 Hvidovre
Denmark
Tel: +45 386 22828
E-mail: tor...@drcmr.dk
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Running dedicated RGWs for async tasks

[Konstantin Shalygin]

Hi,

Yes. You need to turn off gc, lc threads in config for your current (client 
side) RGW's.

Then setup your 'async tasks' RGW without client traffic. No special 
configuration needed, only if I wanna tune gc, lc settings


k
Sent from my iPhone

> On 7 Mar 2024, at 13:09, Marc Singer  wrote:
> 
> Hello Ceph Users
> 
> Since we are running a big S3 cluster we would like to externalize the RGW 
> daemons that do async tasks, like:
> 
> * Garbage collection
> * Lifecycle policies
> * Calculating and updating quotas
> 
> Would this be possible to do in the configuration? Which config values would 
> I need to set on the exposed RGWs and which on the Async Task RGWs?
> 
> Thanks for your input.
> 
> Marc
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: erasure-code-lrc Questions regarding repair

[Ansgar Jazdzewski]

Hi,

I somehow missed your message thanks for your effort to raise this issue

Ansgar

Am Di., 16. Jan. 2024 um 10:05 Uhr schrieb Eugen Block :
>
> Hi,
>
> I don't really have an answer, I just wanted to mention that I created
> a tracker issue [1] because I believe there's a bug in the LRC plugin.
> But there hasn't been any response yet.
>
> [1] https://tracker.ceph.com/issues/61861
>
> Zitat von Ansgar Jazdzewski :
>
> > hi folks,
> >
> > I currently test erasure-code-lrc (1) in a multi-room multi-rack setup.
> > The idea is to be able to repair a disk-failures within the rack
> > itself to lower bandwidth-usage
> >
> > ```bash
> > ceph osd erasure-code-profile set lrc_hdd \
> > plugin=lrc \
> > crush-root=default \
> > crush-locality=rack \
> > crush-failure-domain=host \
> > crush-device-class=hdd \
> > mapping=__D__D__D__D \
> > layers='
> > [
> > [ "_cD_cD_cD_cD", "" ],
> > [ "cDD_", "" ],
> > [ "___cDD__", "" ],
> > [ "__cDD___", "" ],
> > [ "_cDD", "" ],
> > ]' \
> > crush-steps='[
> > [ "choose", "room", 4 ],
> > [ "choose", "rack", 1 ],
> > [ "chooseleaf", "host", 7 ],
> > ]'
> > ```
> >
> > The roule picks 4 out of 5 rooms and keeps the PG in one rack like expected!
> >
> > However it looks like the PG will not move to another Room if the PG
> > is undersized or the entire Room or Rack is down!
> >
> > Questions:
> > * do I miss something to allow LRC (PG's) to move across Racks/Rooms
> > for repair?
> > * Is it even possible to build such a 'Multi-stage' grushmap?
> >
> > Thanks for your help,
> > Ansgar
> >
> > 1) https://docs.ceph.com/en/quincy/rados/operations/erasure-code-jerasure/
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph-storage slack access

[Gregory Farnum]

The slack workspace is bridged to our also-published irc channels. I
don't think we've done anything to enable xmpp (and two protocols is
enough work to keep alive!).
-Greg

On Wed, Mar 6, 2024 at 9:07 AM Marc  wrote:
>
> Is it possible to access this also with xmpp?
>
> >
> > At the very bottom of this page is a link
> > https://ceph.io/en/community/connect/
> >
> > Respectfully,
> >
> > *Wes Dillingham*
> > w...@wesdillingham.com
> > LinkedIn 
> >
> >
> > On Wed, Mar 6, 2024 at 11:45 AM Matthew Vernon 
> > wrote:
> >
> > > Hi,
> > >
> > > How does one get an invite to the ceph-storage slack, please?
> > >
> > > Thanks,
> > >
> > > Matthew
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Minimum amount of nodes needed for stretch mode?

[Stefan Kooman]

Hi,

TL;DR

Failure domain considered is data center. Cluster in stretch mode [1].

- What is the minimum amount of monitor nodes (apart from tie breaker) 
needed per failure domain?


- What is the minimum amount of storage nodes needed per failure domain?

- Are device classes supported with stretch mode?

- is min_size = 1 in "degraded stretch mode" a hard coded requirement or 
can this be changed to it at leave min_size = 2 (yes, I'm aware that no 
other OSD may go down in the surviving data center or PGs will become 
unavailable).



I've converted a (test) 3 node replicated cluster (2 storage nodes, 1 
node with monitor only, min_size=2, size=4) setup to a "stretch mode" 
setup [1]. That works as expected.


CRUSH rule (adjusted to work with 1 host and 2 OSDs per device class per 
data center only)


rule stretch_rule {
id 5
type replicated
step take dc1
step choose firstn 0 type host
step chooseleaf firstn 2 type osd
step emit
step take dc2
step choose firstn 0 type host
step chooseleaf firstn 2 type osd
step emit
}

The documentation seems to suggest that 2 storage nodes and 2 monitor 
nodes are needed at a minimum. Is that correct? I wonder why? For a 
minimal (as possible) cluster I don't see the need for one additional 
monitor per datacenter. Does the tiebreaker monitor function as a normal 
monitor (apart from it not allowed to become leader)?


When stretch rules with device classes are used things don't work as 
expected anymore. Example crush rule:



rule stretch_rule_ssd {
id 4
type replicated
step take dc1 class ssd
step choose firstn 0 type host
step chooseleaf firstn 2 type osd
step emit
step take dc2 class ssd
step choose firstn 0 type host
step chooseleaf firstn 2 type osd
step emit
}

A similar crush rule for hdd exists. When I change the crush_rule for 
one of the pools to use stretch_rule_ssd the PGs on OSDs with device 
class ssd become inactive as soon as one of the data centers goes 
offline (and "degraded stretched mode" has been activated, and only 1 
bucket, data center, is needed for peering). I don't understand why. 
Another issue with this is that as soon as the datacenter is online 
again, the recovery will never finish by itself and a "ceph osd 
force_healthy_stretch_mode --yes-i-really-mean-it" is needed to get 
HEALTH_OK


Can anyone explain to me why this is?

Gr. Stefan

[1]: https://docs.ceph.com/en/latest/rados/operations/stretch-mode/

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Minimum amount of nodes needed for stretch mode?

[Gregory Farnum]

On Thu, Mar 7, 2024 at 9:09 AM Stefan Kooman  wrote:
>
> Hi,
>
> TL;DR
>
> Failure domain considered is data center. Cluster in stretch mode [1].
>
> - What is the minimum amount of monitor nodes (apart from tie breaker)
> needed per failure domain?

You need at least two monitors per site. This is because in stretch
mode, OSDs can *only* connect to monitors in their data center. You
don't want a restarting monitor to lead to the entire cluster
switching to degraded stretch mode.

>
> - What is the minimum amount of storage nodes needed per failure domain?

I don't think there's anything specific here beyond supporting 2
replicas per site (so, at least two OSD hosts unless you configure
things weirdly).

> - Are device classes supported with stretch mode?

Stretch mode doesn't have any specific logic around device classes, so
if you provide CRUSH rules that work with them it should be okay? But
definitely not tested.

> - is min_size = 1 in "degraded stretch mode" a hard coded requirement or
> can this be changed to it at leave min_size = 2 (yes, I'm aware that no
> other OSD may go down in the surviving data center or PGs will become
> unavailable).

Hmm, I'm pretty sure this is hard coded but haven't checked in a
while. Hopefully somebody else can chime in here.

> I've converted a (test) 3 node replicated cluster (2 storage nodes, 1
> node with monitor only, min_size=2, size=4) setup to a "stretch mode"
> setup [1]. That works as expected.
>
> CRUSH rule (adjusted to work with 1 host and 2 OSDs per device class per
> data center only)
>
> rule stretch_rule {
> id 5
> type replicated
> step take dc1
> step choose firstn 0 type host
> step chooseleaf firstn 2 type osd
> step emit
> step take dc2
> step choose firstn 0 type host
> step chooseleaf firstn 2 type osd
> step emit
> }
>
> The documentation seems to suggest that 2 storage nodes and 2 monitor
> nodes are needed at a minimum. Is that correct? I wonder why? For a
> minimal (as possible) cluster I don't see the need for one additional
> monitor per datacenter. Does the tiebreaker monitor function as a normal
> monitor (apart from it not allowed to become leader)?
>
> When stretch rules with device classes are used things don't work as
> expected anymore. Example crush rule:
>
>
> rule stretch_rule_ssd {
> id 4
> type replicated
> step take dc1 class ssd
> step choose firstn 0 type host
> step chooseleaf firstn 2 type osd
> step emit
> step take dc2 class ssd
> step choose firstn 0 type host
> step chooseleaf firstn 2 type osd
> step emit
> }
>
> A similar crush rule for hdd exists. When I change the crush_rule for
> one of the pools to use stretch_rule_ssd the PGs on OSDs with device
> class ssd become inactive as soon as one of the data centers goes
> offline (and "degraded stretched mode" has been activated, and only 1
> bucket, data center, is needed for peering). I don't understand why.
> Another issue with this is that as soon as the datacenter is online
> again, the recovery will never finish by itself and a "ceph osd
> force_healthy_stretch_mode --yes-i-really-mean-it" is needed to get
> HEALTH_OK

Hmm. My first guess is there's something going on with the shadow
CRUSH tree from device classes, but somebody would need to dig into
that. I think you should file a ticket.
-Greg

>
> Can anyone explain to me why this is?
>
> Gr. Stefan
>
> [1]: https://docs.ceph.com/en/latest/rados/operations/stretch-mode/
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] All MGR loop crash

[David C.]

Hello everybody,

I'm encountering strange behavior on an infrastructure (it's pre-production
but it's very ugly). After a "drain" on monitor (and a manager). MGRs all
crash on startup:

Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr ms_dispatch2 standby
mgrmap(e 1310) v1
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map received
map epoch 1310
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map active in
map: 1 active is 99148504
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map Activating!
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map I am now
activating
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr ms_dispatch2 mgrmap(e
1310) v1
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgrc handle_mgr_map Got map
version 1310
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgrc handle_mgr_map Active mgr
is now
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgrc reconnect No active mgr
available yet
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr init waiting for OSDMap...
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: monclient: _renew_subs
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: monclient: _send_mon_message
to mon.idf-pprod-osd3 at v2:X.X.X.X:3300/0 
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: monclient: _reopen_session
rank -1
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: *** Caught signal (Aborted) **
  in thread 7f9a07a27640
thread_name:mgr-fin

  ceph version
17.2.6-170.el9cp (59bbeb8815ec3aeb3c8bba1e1866f8f6729eb840) quincy (stable)
  1:
/lib64/libc.so.6(+0x54db0) [0x7f9a2364ddb0]
  2:
/lib64/libc.so.6(+0xa154c) [0x7f9a2369a54c]
  3: raise()
  4: abort()
  5:
/usr/lib64/ceph/libceph-common.so.2(+0x1c1fa8) [0x7f9a23ce2fa8]
  6:
/usr/lib64/ceph/libceph-common.so.2(+0x25) [0x7f9a23f65425]
  7:
/usr/lib64/ceph/libceph-common.so.2(+0x4442e0) [0x7f9a23f652e0]
  8:
/usr/lib64/ceph/libceph-common.so.2(+0x4442e0) [0x7f9a23f652e0]
  9:
(MonClient::_add_conns()+0x242) [0x7f9a23f5fa42]
  10:
(MonClient::_reopen_session(int)+0x428) [0x7f9a23f60518]
  11: (Mgr::init()+0x384)
[0x5604667a6434]
  12:
/usr/bin/ceph-mgr(+0x1af271) [0x5604667ae271]
  13:
/usr/bin/ceph-mgr(+0x11364d) [0x56046671264d]
  14:
(Finisher::finisher_thread_entry()+0x175) [0x7f9a23d10645]
  15:
/lib64/libc.so.6(+0x9f802) [0x7f9a23698802]
  16:
/lib64/libc.so.6(+0x3f450) [0x7f9a23638450]
  NOTE: a copy of the
executable, or `objdump -rdS ` is needed to interpret this.

I have the impression that the MGRs are ejected by the monitors, however
after debugging monitor, I don't see anything abnormal on the monitor side
(if I haven't missed something).

All we can see is that we get an exception on the "_add_conn" method (
https://github.com/ceph/ceph/blob/v17.2.6/src/mon/MonClient.cc #L775)

Version : 17.2.6-170.el9cp (RHCS6)
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph-storage slack access

[Marc]

What is this irc access then? Is there some webclient that can be used? Is this 
ceph.io down? Can't get a website nor a ping.


> 
> The slack workspace is bridged to our also-published irc channels. I
> don't think we've done anything to enable xmpp (and two protocols is
> enough work to keep alive!).
> -Greg
> 
> On Wed, Mar 6, 2024 at 9:07 AM Marc  wrote:
> >
> > Is it possible to access this also with xmpp?
> >
> > >
> > > At the very bottom of this page is a link
> > > https://ceph.io/en/community/connect/
> > >
> > > Respectfully,
> > >
> > > *Wes Dillingham*
> > > w...@wesdillingham.com
> > > LinkedIn 
> > >
> > >
> > > On Wed, Mar 6, 2024 at 11:45 AM Matthew Vernon 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > How does one get an invite to the ceph-storage slack, please?
> > > >
> > > > Thanks,
> > > >
> > > > Matthew

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: All MGR loop crash

[David C.]

I took the wrong ligne =>
https://github.com/ceph/ceph/blob/v17.2.6/src/mon/MonClient.cc#L822


Le jeu. 7 mars 2024 à 18:21, David C.  a écrit :

>
> Hello everybody,
>
> I'm encountering strange behavior on an infrastructure (it's
> pre-production but it's very ugly). After a "drain" on monitor (and a
> manager). MGRs all crash on startup:
>
> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr ms_dispatch2 standby
> mgrmap(e 1310) v1
> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map received
> map epoch 1310
> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map active in
> map: 1 active is 99148504
> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map Activating!
> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map I am now
> activating
> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr ms_dispatch2 mgrmap(e
> 1310) v1
> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgrc handle_mgr_map Got map
> version 1310
> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgrc handle_mgr_map Active
> mgr is now
> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgrc reconnect No active mgr
> available yet
> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr init waiting for OSDMap...
> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: monclient: _renew_subs
> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: monclient: _send_mon_message
> to mon.idf-pprod-osd3 at v2:X.X.X.X:3300/0 
> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: monclient: _reopen_session
> rank -1
> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: *** Caught signal (Aborted) **
>   in thread 7f9a07a27640
> thread_name:mgr-fin
>
>   ceph version
> 17.2.6-170.el9cp (59bbeb8815ec3aeb3c8bba1e1866f8f6729eb840) quincy (stable)
>   1:
> /lib64/libc.so.6(+0x54db0) [0x7f9a2364ddb0]
>   2:
> /lib64/libc.so.6(+0xa154c) [0x7f9a2369a54c]
>   3: raise()
>   4: abort()
>   5:
> /usr/lib64/ceph/libceph-common.so.2(+0x1c1fa8) [0x7f9a23ce2fa8]
>   6:
> /usr/lib64/ceph/libceph-common.so.2(+0x25) [0x7f9a23f65425]
>   7:
> /usr/lib64/ceph/libceph-common.so.2(+0x4442e0) [0x7f9a23f652e0]
>   8:
> /usr/lib64/ceph/libceph-common.so.2(+0x4442e0) [0x7f9a23f652e0]
>   9:
> (MonClient::_add_conns()+0x242) [0x7f9a23f5fa42]
>   10:
> (MonClient::_reopen_session(int)+0x428) [0x7f9a23f60518]
>   11: (Mgr::init()+0x384)
> [0x5604667a6434]
>   12:
> /usr/bin/ceph-mgr(+0x1af271) [0x5604667ae271]
>   13:
> /usr/bin/ceph-mgr(+0x11364d) [0x56046671264d]
>   14:
> (Finisher::finisher_thread_entry()+0x175) [0x7f9a23d10645]
>   15:
> /lib64/libc.so.6(+0x9f802) [0x7f9a23698802]
>   16:
> /lib64/libc.so.6(+0x3f450) [0x7f9a23638450]
>   NOTE: a copy of the
> executable, or `objdump -rdS ` is needed to interpret this.
>
> I have the impression that the MGRs are ejected by the monitors, however
> after debugging monitor, I don't see anything abnormal on the monitor side
> (if I haven't missed something).
>
> All we can see is that we get an exception on the "_add_conn" method (
> https://github.com/ceph/ceph/blob/v17.2.6/src/mon/MonClient.cc #L775)
>
> Version : 17.2.6-170.el9cp (RHCS6)
>
>
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: All MGR loop crash

[David C.]

Ok, got it :

[root@pprod-admin:/var/lib/ceph/]# ceph mon dump -f json-pretty
|egrep "name|weigh"
dumped monmap epoch 14
"min_mon_release_name": "quincy",
"name": "pprod-mon2",
"weight": 10,
"name": "pprod-mon3",
"weight": 10,
"name": "pprod-osd2",
"weight": 0,
"name": "pprod-osd1",
"weight": 0,
"name": "pprod-osd3",
"weight": 0,

ceph mon set-weight pprod-mon2 0
ceph mon set-weight pprod-mon3 0

And restart ceph-mgr

Le jeu. 7 mars 2024 à 18:25, David C.  a écrit :

> I took the wrong ligne =>
> https://github.com/ceph/ceph/blob/v17.2.6/src/mon/MonClient.cc#L822
>
>
> Le jeu. 7 mars 2024 à 18:21, David C.  a écrit :
>
>>
>> Hello everybody,
>>
>> I'm encountering strange behavior on an infrastructure (it's
>> pre-production but it's very ugly). After a "drain" on monitor (and a
>> manager). MGRs all crash on startup:
>>
>> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr ms_dispatch2 standby
>> mgrmap(e 1310) v1
>> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map received
>> map epoch 1310
>> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map active in
>> map: 1 active is 99148504
>> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map
>> Activating!
>> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map I am now
>> activating
>> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr ms_dispatch2 mgrmap(e
>> 1310) v1
>> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgrc handle_mgr_map Got map
>> version 1310
>> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgrc handle_mgr_map Active
>> mgr is now
>> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgrc reconnect No active mgr
>> available yet
>> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr init waiting for
>> OSDMap...
>> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: monclient: _renew_subs
>> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: monclient: _send_mon_message
>> to mon.idf-pprod-osd3 at v2:X.X.X.X:3300/0 
>> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: monclient: _reopen_session
>> rank -1
>> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: *** Caught signal (Aborted)
>> **
>>   in thread 7f9a07a27640
>> thread_name:mgr-fin
>>
>>   ceph version
>> 17.2.6-170.el9cp (59bbeb8815ec3aeb3c8bba1e1866f8f6729eb840) quincy (stable)
>>   1:
>> /lib64/libc.so.6(+0x54db0) [0x7f9a2364ddb0]
>>   2:
>> /lib64/libc.so.6(+0xa154c) [0x7f9a2369a54c]
>>   3: raise()
>>   4: abort()
>>   5:
>> /usr/lib64/ceph/libceph-common.so.2(+0x1c1fa8) [0x7f9a23ce2fa8]
>>   6:
>> /usr/lib64/ceph/libceph-common.so.2(+0x25) [0x7f9a23f65425]
>>   7:
>> /usr/lib64/ceph/libceph-common.so.2(+0x4442e0) [0x7f9a23f652e0]
>>   8:
>> /usr/lib64/ceph/libceph-common.so.2(+0x4442e0) [0x7f9a23f652e0]
>>   9:
>> (MonClient::_add_conns()+0x242) [0x7f9a23f5fa42]
>>   10:
>> (MonClient::_reopen_session(int)+0x428) [0x7f9a23f60518]
>>   11: (Mgr::init()+0x384)
>> [0x5604667a6434]
>>   12:
>> /usr/bin/ceph-mgr(+0x1af271) [0x5604667ae271]
>>   13:
>> /usr/bin/ceph-mgr(+0x11364d) [0x56046671264d]
>>   14:
>> (Finisher::finisher_thread_entry()+0x175) [0x7f9a23d10645]
>>   15:
>> /lib64/libc.so.6(+0x9f802) [0x7f9a23698802]
>>   16:
>> /lib64/libc.so.6(+0x3f450) [0x7f9a23638450]
>>   NOTE: a copy of the
>> executable, or `objdump -rdS ` is needed to interpret this.
>>
>> I have the impression that the MGRs are ejected by the monitors, however
>> after debugging monitor, I don't see anything abnormal on the monitor side
>> (if I haven't missed something).
>>
>> All we can see is that we get an exception on the "_add_conn" method (
>> https://github.com/ceph/ceph/blob/v17.2.6/src/mon/MonClient.cc #L775)
>>
>> Version : 17.2.6-170.el9cp (RHCS6)
>>
>>
>>
>>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: All MGR loop crash

[Eugen Block]

I’m curious how the weights might have been changed. I’ve never  
touched a mon weight myself, do you know how that happened?


Zitat von "David C." :


Ok, got it :

[root@pprod-admin:/var/lib/ceph/]# ceph mon dump -f json-pretty
|egrep "name|weigh"
dumped monmap epoch 14
"min_mon_release_name": "quincy",
"name": "pprod-mon2",
"weight": 10,
"name": "pprod-mon3",
"weight": 10,
"name": "pprod-osd2",
"weight": 0,
"name": "pprod-osd1",
"weight": 0,
"name": "pprod-osd3",
"weight": 0,

ceph mon set-weight pprod-mon2 0
ceph mon set-weight pprod-mon3 0

And restart ceph-mgr

Le jeu. 7 mars 2024 à 18:25, David C.  a écrit :


I took the wrong ligne =>
https://github.com/ceph/ceph/blob/v17.2.6/src/mon/MonClient.cc#L822


Le jeu. 7 mars 2024 à 18:21, David C.  a écrit :



Hello everybody,

I'm encountering strange behavior on an infrastructure (it's
pre-production but it's very ugly). After a "drain" on monitor (and a
manager). MGRs all crash on startup:

Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr ms_dispatch2 standby
mgrmap(e 1310) v1
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map received
map epoch 1310
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map active in
map: 1 active is 99148504
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map
Activating!
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map I am now
activating
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr ms_dispatch2 mgrmap(e
1310) v1
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgrc handle_mgr_map Got map
version 1310
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgrc handle_mgr_map Active
mgr is now
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgrc reconnect No active mgr
available yet
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr init waiting for
OSDMap...
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: monclient: _renew_subs
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: monclient: _send_mon_message
to mon.idf-pprod-osd3 at v2:X.X.X.X:3300/0 
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: monclient: _reopen_session
rank -1
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: *** Caught signal (Aborted)
**
  in thread 7f9a07a27640
thread_name:mgr-fin

  ceph version
17.2.6-170.el9cp (59bbeb8815ec3aeb3c8bba1e1866f8f6729eb840) quincy (stable)
  1:
/lib64/libc.so.6(+0x54db0) [0x7f9a2364ddb0]
  2:
/lib64/libc.so.6(+0xa154c) [0x7f9a2369a54c]
  3: raise()
  4: abort()
  5:
/usr/lib64/ceph/libceph-common.so.2(+0x1c1fa8) [0x7f9a23ce2fa8]
  6:
/usr/lib64/ceph/libceph-common.so.2(+0x25) [0x7f9a23f65425]
  7:
/usr/lib64/ceph/libceph-common.so.2(+0x4442e0) [0x7f9a23f652e0]
  8:
/usr/lib64/ceph/libceph-common.so.2(+0x4442e0) [0x7f9a23f652e0]
  9:
(MonClient::_add_conns()+0x242) [0x7f9a23f5fa42]
  10:
(MonClient::_reopen_session(int)+0x428) [0x7f9a23f60518]
  11: (Mgr::init()+0x384)
[0x5604667a6434]
  12:
/usr/bin/ceph-mgr(+0x1af271) [0x5604667ae271]
  13:
/usr/bin/ceph-mgr(+0x11364d) [0x56046671264d]
  14:
(Finisher::finisher_thread_entry()+0x175) [0x7f9a23d10645]
  15:
/lib64/libc.so.6(+0x9f802) [0x7f9a23698802]
  16:
/lib64/libc.so.6(+0x3f450) [0x7f9a23638450]
  NOTE: a copy of the
executable, or `objdump -rdS ` is needed to interpret this.

I have the impression that the MGRs are ejected by the monitors, however
after debugging monitor, I don't see anything abnormal on the monitor side
(if I haven't missed something).

All we can see is that we get an exception on the "_add_conn" method (
https://github.com/ceph/ceph/blob/v17.2.6/src/mon/MonClient.cc #L775)

Version : 17.2.6-170.el9cp (RHCS6)





___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Minimum amount of nodes needed for stretch mode?

[Stefan Kooman]

On 07-03-2024 18:16, Gregory Farnum wrote:

On Thu, Mar 7, 2024 at 9:09 AM Stefan Kooman  wrote:


Hi,

TL;DR

Failure domain considered is data center. Cluster in stretch mode [1].

- What is the minimum amount of monitor nodes (apart from tie breaker)
needed per failure domain?


You need at least two monitors per site. This is because in stretch
mode, OSDs can *only* connect to monitors in their data center. You
don't want a restarting monitor to lead to the entire cluster
switching to degraded stretch mode.


It does not seem to work like that. I mean, it's true that OSDs within 
that data center cannot peer or be restarted when the monitor is 
unavailable, but it can be offline without leading to unavailability. 
When I stop a monitor (does not matter which one) the cluster stays 
online. If one of the monitors in a data center is offline, OSDs in that 
datacenter that get restarted won't be able to come online anymore (as 
expected). Only when the whole data center is offline will stretch mode 
be triggered. The docs are pretty clear about this, and it's exactly 
what I see in practice:


"If all OSDs and monitors in one of the data centers become inaccessible 
at once, the surviving data center enters a “degraded stretch mode”."


It does not have to be "all at once" though. When the last remaining 
daemon in a failure domain goes offline degraded stretch mode gets 
triggered. Offline all OSDs and keep the monitor(s) alive? 100% inactive 
PGs as the "peer from two buckets" enforcement rule is still active (as 
designed).







- What is the minimum amount of storage nodes needed per failure domain?


I don't think there's anything specific here beyond supporting 2
replicas per site (so, at least two OSD hosts unless you configure
things weirdly).


lol. I personally don't think its weird: it's the counterpart of the 
minimal replicated 3-node cluster.





- Are device classes supported with stretch mode?


Stretch mode doesn't have any specific logic around device classes, so
if you provide CRUSH rules that work with them it should be okay? But
definitely not tested.


It does not seem to work. But I'll repeat the process once more and 
double check.





- is min_size = 1 in "degraded stretch mode" a hard coded requirement or
can this be changed to it at leave min_size = 2 (yes, I'm aware that no
other OSD may go down in the surviving data center or PGs will become
unavailable).


Hmm, I'm pretty sure this is hard coded but haven't checked in a
while. Hopefully somebody else can chime in here.


I would like this to be configurable to be honest. As it's a policy 
decision and not a technical requirement. Non stretch mode clusters also 
have the freedom of choice. If the operator values data consistency over 
data availability it should be honored.





I've converted a (test) 3 node replicated cluster (2 storage nodes, 1
node with monitor only, min_size=2, size=4) setup to a "stretch mode"
setup [1]. That works as expected.

CRUSH rule (adjusted to work with 1 host and 2 OSDs per device class per
data center only)

rule stretch_rule {
 id 5
 type replicated
 step take dc1
 step choose firstn 0 type host
 step chooseleaf firstn 2 type osd
 step emit
 step take dc2
 step choose firstn 0 type host
 step chooseleaf firstn 2 type osd
 step emit
}

The documentation seems to suggest that 2 storage nodes and 2 monitor
nodes are needed at a minimum. Is that correct? I wonder why? For a
minimal (as possible) cluster I don't see the need for one additional
monitor per datacenter. Does the tiebreaker monitor function as a normal
monitor (apart from it not allowed to become leader)?

When stretch rules with device classes are used things don't work as
expected anymore. Example crush rule:


rule stretch_rule_ssd {
 id 4
 type replicated
 step take dc1 class ssd
 step choose firstn 0 type host
 step chooseleaf firstn 2 type osd
 step emit
 step take dc2 class ssd
 step choose firstn 0 type host
 step chooseleaf firstn 2 type osd
 step emit
}

A similar crush rule for hdd exists. When I change the crush_rule for
one of the pools to use stretch_rule_ssd the PGs on OSDs with device
class ssd become inactive as soon as one of the data centers goes
offline (and "degraded stretched mode" has been activated, and only 1
bucket, data center, is needed for peering). I don't understand why.
Another issue with this is that as soon as the datacenter is online
again, the recovery will never finish by itself and a "ceph osd
force_healthy_stretch_mode --yes-i-really-mean-it" is needed to get
HEALTH_OK


Hmm. My first guess is there's something going on with the shadow
CRUSH tree from device classes, but somebody would need to dig into
that. I think you should file a ticket.


I will do some more testing and file a ticket if I'm confident it should 
work but doesn't.

[ceph-users] Re: Disable signature url in ceph rgw

[Casey Bodley]

anything we can do to narrow down the policy issue here? any of the
Principal, Action, Resource, or Condition matches could be failing
here. you might try replacing each with a wildcard, one at a time,
until you see the policy take effect

On Wed, Dec 13, 2023 at 5:04 AM Marc Singer  wrote:
>
> Hi
>
> As my attachment is very messy, I cleaned it up and provide a much
> simpler version for your tests bellow.
> These policies seem to get ignored when the URL is presigned.
>
> {
> "Version":"2012-10-17",
> "Id":"userbucket%%%policy",
> "Statement":[
>{
>   "Sid":"username%%%read",
>   "Effect":"Allow",
>   "Principal":{
>  "AWS":"arn:aws:iam:::user/username"
>   },
>   "Action":[
>  "s3:ListBucket",
>  "s3:ListBucketVersions",
>  "s3:GetObject",
>  "s3:GetObjectVersion"
>   ],
>   "Resource":[
>  "arn:aws:s3:::userbucket",
>  "arn:aws:s3:::userbucket/*"
>   ],
>   "Condition":{
>  "IpAddress":{
> "aws:SourceIp":[
>"redacted"
> ]
>  }
>   }
>},
>{
>   "Sid":"username%%%write",
>   "Effect":"Allow",
>   "Principal":{
>  "AWS":"arn:aws:iam:::user/username"
>   },
>   "Action":[
>  "s3:PutObject",
>  "s3:DeleteObject",
>  "s3:DeleteObjectVersion",
>  "s3:ListBucketMultipartUploads",
>  "s3:ListMultipartUploadParts",
>  "s3:AbortMultipartUpload"
>   ],
>   "Resource":[
>  "arn:aws:s3:::userbucket",
>  "arn:aws:s3:::userbucket/*"
>   ],
>   "Condition":{
>  "IpAddress":{
> "aws:SourceIp":[
>"redacted"
> ]
>  }
>   }
>},
>{
>   "Sid":"username%%%policy_control",
>   "Effect":"Deny",
>   "Principal":{
>  "AWS":"arn:aws:iam:::user/username"
>   },
>   "Action":[
>  "s3:PutObjectAcl",
>  "s3:GetObjectAcl",
>  "s3:PutBucketAcl",
>  "s3:GetBucketPolicy",
>  "s3:DeleteBucketPolicy",
>  "s3:PutBucketPolicy"
>   ],
>   "Resource":[
>  "arn:aws:s3:::userbucket",
>  "arn:aws:s3:::userbucket/*"
>   ]
>}
> ]
> }
>
> Thanks and yours sincerely
>
> Marc Singer
>
> On 2023-12-12 10:24, Marc Singer wrote:
> > Hi
> >
> > First, all requests with presigned URLs should be restricted.
> >
> > This is how the request is blocked with the nginx sidecar (it's just a
> > simple parameter in the URL that is forbidden):
> >
> > if ($arg_Signature) { return 403 'Signature parameter forbidden';
> >
> > }
> >
> > Our bucket policies are created automatically with a custom
> > microservice. You find an example in attachment from a random "managed"
> > bucket. These buckets are affected by the issue.
> >
> > There is a policy that stops users from changing the policy.
> >
> > I might have done a mistake when redacting replacing a user with the
> > same values.
> >
> > Thanks you and have a great day
> >
> > Marc
> >
> > On 12/9/23 00:37, Robin H. Johnson wrote:
> >> On Fri, Dec 08, 2023 at 10:41:59AM +0100,marc@singer.services  wrote:
> >>> Hi Ceph users
> >>>
> >>> We are using Ceph Pacific (16) in this specific deployment.
> >>>
> >>> In our use case we do not want our users to be able to generate
> >>> signature v4 URLs because they bypass the policies that we set on
> >>> buckets (e.g IP restrictions).
> >>> Currently we have a sidecar reverse proxy running that filters
> >>> requests with signature URL specific request parameters.
> >>> This is obviously not very efficient and we are looking to replace
> >>> this somehow in the future.
> >>>
> >>> 1. Is there an option in RGW to disable this signed URLs (e.g
> >>> returning status 403)?
> >>> 2. If not is this planned or would it make sense to add it as a
> >>> configuration option?
> >>> 3. Or is the behaviour of not respecting bucket policies in RGW with
> >>> signature v4 URLs a bug and they should be actually applied?
> >> Trying to clarify your ask:
> >> - you want ALL requests, including presigned URLs, to be subject to
> >> the
> >>IP restrictions encoded in your bucket policy?
> >>e.g. auth (signature AND IP-list)
> >>
> >> That should be possible with bucket policy.
> >>
> >> Can you post the current bucket policy that you have? (redact with
> >> distinct values the IPs, userids, bucket name, any paths, but
> >> otherwise
> >> keep it complete).
> >>
> >> You cannot fundamentally stop anybody from generating presigned URLs,
> >> because that's purely a client-side operation. Generating presigned
> >> URLs
> >> requires an access key and secret key, a

[ceph-users] Re: Remove cluster_network without routing

[Anthony D'Atri]

I think heartbeats will failover to the public network if the private doesn't 
work -- may not have always done that.



>> Hi
>> Cephadm Reef 18.2.0.
>> We would like to remove our cluster_network without stopping the cluster and 
>> without having to route between the networks.
>> globaladvanced  cluster_network192.168.100.0/24  
>>   *
>> globaladvanced  public_network172.21.12.0/22 
>>*
>> The documentation[1] states:
>> "
>> You may specifically assign static IP addresses or override cluster_network 
>> settings using the cluster_addr setting for specific OSD daemons.
>> "
>> So for one OSD at a time I could set cluster_addr to override the 
>> cluster_network IP and use the public_network IP instead? As the containers 
>> are using host networking they have access to both IPs and will just layer 2 
>> the traffic, avoiding routing?
>> When all OSDs are running with a public_network IP set via cluster_addr we 
>> can just delete the cluster_network setting and then remove all the 
>> ceph_addr settings, as with no cluster_network setting the public_network 
>> setting will be used?
>> We tried with one OSD and it seems to work. Anyone see a problem with this 
>> approach?
> 
> Turned out to be even simpler for this setup where the OSD containers have 
> access til both host networks:
> 
> 1) ceph config rm global cluster_network -> nothing happened, no automatic 
> redeploy or restart
> 
> 2) Restart OSDs
> 
> Mvh.
> 
> Torkil
> 
>> Thanks
>> Torkil
>> [1] 
>> https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#id3
> 
> -- 
> Torkil Svensgaard
> Sysadmin
> MR-Forskningssektionen, afs. 714
> DRCMR, Danish Research Centre for Magnetic Resonance
> Hvidovre Hospital
> Kettegård Allé 30
> DK-2650 Hvidovre
> Denmark
> Tel: +45 386 22828
> E-mail: tor...@drcmr.dk
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] All MGR loop crash

[David C.]

some monitors have existed for many years (weight 10) others have been
added (weight 0)

=> https://github.com/ceph/ceph/commit/2d113dedf851995e000d3cce136b69
bfa94b6fe0

Le jeudi 7 mars 2024, Eugen Block  a écrit :

> I’m curious how the weights might have been changed. I’ve never touched a
> mon weight myself, do you know how that happened?
>
> Zitat von "David C." :
>
> Ok, got it :
>>
>> [root@pprod-admin:/var/lib/ceph/]# ceph mon dump -f json-pretty
>> |egrep "name|weigh"
>> dumped monmap epoch 14
>> "min_mon_release_name": "quincy",
>> "name": "pprod-mon2",
>> "weight": 10,
>> "name": "pprod-mon3",
>> "weight": 10,
>> "name": "pprod-osd2",
>> "weight": 0,
>> "name": "pprod-osd1",
>> "weight": 0,
>> "name": "pprod-osd3",
>> "weight": 0,
>>
>> ceph mon set-weight pprod-mon2 0
>> ceph mon set-weight pprod-mon3 0
>>
>> And restart ceph-mgr
>>
>> Le jeu. 7 mars 2024 à 18:25, David C.  a écrit :
>>
>> I took the wrong ligne =>
>>> https://github.com/ceph/ceph/blob/v17.2.6/src/mon/MonClient.cc#L822
>>>
>>>
>>> Le jeu. 7 mars 2024 à 18:21, David C.  a écrit :
>>>
>>>
 Hello everybody,

 I'm encountering strange behavior on an infrastructure (it's
 pre-production but it's very ugly). After a "drain" on monitor (and a
 manager). MGRs all crash on startup:

 Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr ms_dispatch2 standby
 mgrmap(e 1310) v1
 Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map received
 map epoch 1310
 Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map active
 in
 map: 1 active is 99148504
 Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map
 Activating!
 Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map I am now
 activating
 Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr ms_dispatch2 mgrmap(e
 1310) v1
 Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgrc handle_mgr_map Got map
 version 1310
 Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgrc handle_mgr_map Active
 mgr is now
 Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgrc reconnect No active
 mgr
 available yet
 Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr init waiting for
 OSDMap...
 Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: monclient: _renew_subs
 Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: monclient:
 _send_mon_message
 to mon.idf-pprod-osd3 at v2:X.X.X.X:3300/0 
 Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: monclient: _reopen_session
 rank -1
 Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: *** Caught signal (Aborted)
 **
   in thread 7f9a07a27640
 thread_name:mgr-fin

   ceph version
 17.2.6-170.el9cp (59bbeb8815ec3aeb3c8bba1e1866f8f6729eb840) quincy
 (stable)
   1:
 /lib64/libc.so.6(+0x54db0) [0x7f9a2364ddb0]
   2:
 /lib64/libc.so.6(+0xa154c) [0x7f9a2369a54c]
   3: raise()
   4: abort()
   5:
 /usr/lib64/ceph/libceph-common.so.2(+0x1c1fa8) [0x7f9a23ce2fa8]
   6:
 /usr/lib64/ceph/libceph-common.so.2(+0x25) [0x7f9a23f65425]
   7:
 /usr/lib64/ceph/libceph-common.so.2(+0x4442e0) [0x7f9a23f652e0]
   8:
 /usr/lib64/ceph/libceph-common.so.2(+0x4442e0) [0x7f9a23f652e0]
   9:
 (MonClient::_add_conns()+0x242) [0x7f9a23f5fa42]
   10:
 (MonClient::_reopen_session(int)+0x428) [0x7f9a23f60518]
   11:
 (Mgr::init()+0x384)
 [0x5604667a6434]
   12:
 /usr/bin/ceph-mgr(+0x1af271) [0x5604667ae271]
   13:
 /usr/bin/ceph-mgr(+0x11364d) [0x56046671264d]
   14:
 (Finisher::finisher_thread_entry()+0x175) [0x7f9a23d10645]
   15:
 /lib64/libc.so.6(+0x9f802) [0x7f9a23698802]
   16:
 /lib64/libc.so.6(+0x3f450) [0x7f9a23638450]
   NOTE: a copy of the
 executable, or `objdump -rdS ` is needed to interpret this.

 I have the impression that the MGRs are ejecte

[ceph-users] Announcing Ceph Day NYC 2024 - April 26th!

[Dan van der Ster]

Hi everyone,

Ceph Days are coming to New York City again this year, co-hosted by
Bloomberg Engineering and Clyso!

We're planning a full day of Ceph content, well timed to learn about the
latest and greatest Squid release.

https://ceph.io/en/community/events/2024/ceph-days-nyc/

We're opening the CFP for presenters today -- it will close on March 26th
so please get your proposals in quickly!

Registration is also open now and space is limited, so book now to reserve
your seat!

Looking forward to seeing you there!

-- dan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph-volume fails when adding spearate DATA and DATA.DB volumes

[service . plant]

Oh, dude! You opened my eyes! I thought (it is written this way in 
documentation) that all the commands need to be executed under cephadm shell.
That is why I always ran 'cephadm shell' first, falling down into container 
env, and then all the rest.
Where can I read about proper usage of cephadm tool?

Thanks in advance!
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Announcing Ceph Day NYC 2024 - April 26th!

[nuabo tan]

good news
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] rgw dynamic bucket sharding will hang io

[nuabo tan]

When reshard occurs, io will be blocked, why has this serious problem not been 
solved?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: All MGR loop crash

[Dieter Roels]

Public

We ran into this issue last week when upgrading to quincy.  We asked ourselves 
the same question: how did the weight change, as we did not even know that was 
a thing.

We checked our other clusters and we have some where all the mons have a weight 
of 10, and there it is not an issue.  So only certain combinations of weights 
cause this crash.

regards,

Dieter Roels

-Original Message-
From: Eugen Block 
Sent: Thursday, 7 March 2024 19:13
To: ceph-users@ceph.io
Subject: [ceph-users] Re: All MGR loop crash



I’m curious how the weights might have been changed. I’ve never touched a mon 
weight myself, do you know how that happened?



Disclaimer 
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: All MGR loop crash

[Eugen Block]

Thanks! That's very interesting to know!

Zitat von "David C." :


some monitors have existed for many years (weight 10) others have been
added (weight 0)

=> https://github.com/ceph/ceph/commit/2d113dedf851995e000d3cce136b69
bfa94b6fe0

Le jeudi 7 mars 2024, Eugen Block  a écrit :


I’m curious how the weights might have been changed. I’ve never touched a
mon weight myself, do you know how that happened?

Zitat von "David C." :

Ok, got it :


[root@pprod-admin:/var/lib/ceph/]# ceph mon dump -f json-pretty
|egrep "name|weigh"
dumped monmap epoch 14
"min_mon_release_name": "quincy",
"name": "pprod-mon2",
"weight": 10,
"name": "pprod-mon3",
"weight": 10,
"name": "pprod-osd2",
"weight": 0,
"name": "pprod-osd1",
"weight": 0,
"name": "pprod-osd3",
"weight": 0,

ceph mon set-weight pprod-mon2 0
ceph mon set-weight pprod-mon3 0

And restart ceph-mgr

Le jeu. 7 mars 2024 à 18:25, David C.  a écrit :

I took the wrong ligne =>

https://github.com/ceph/ceph/blob/v17.2.6/src/mon/MonClient.cc#L822


Le jeu. 7 mars 2024 à 18:21, David C.  a écrit :



Hello everybody,

I'm encountering strange behavior on an infrastructure (it's
pre-production but it's very ugly). After a "drain" on monitor (and a
manager). MGRs all crash on startup:

Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr ms_dispatch2 standby
mgrmap(e 1310) v1
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map received
map epoch 1310
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map active
in
map: 1 active is 99148504
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map
Activating!
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map I am now
activating
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr ms_dispatch2 mgrmap(e
1310) v1
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgrc handle_mgr_map Got map
version 1310
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgrc handle_mgr_map Active
mgr is now
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgrc reconnect No active
mgr
available yet
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr init waiting for
OSDMap...
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: monclient: _renew_subs
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: monclient:
_send_mon_message
to mon.idf-pprod-osd3 at v2:X.X.X.X:3300/0 
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: monclient: _reopen_session
rank -1
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: *** Caught signal (Aborted)
**
  in thread 7f9a07a27640
thread_name:mgr-fin

  ceph version
17.2.6-170.el9cp (59bbeb8815ec3aeb3c8bba1e1866f8f6729eb840) quincy
(stable)
  1:
/lib64/libc.so.6(+0x54db0) [0x7f9a2364ddb0]
  2:
/lib64/libc.so.6(+0xa154c) [0x7f9a2369a54c]
  3: raise()
  4: abort()
  5:
/usr/lib64/ceph/libceph-common.so.2(+0x1c1fa8) [0x7f9a23ce2fa8]
  6:
/usr/lib64/ceph/libceph-common.so.2(+0x25) [0x7f9a23f65425]
  7:
/usr/lib64/ceph/libceph-common.so.2(+0x4442e0) [0x7f9a23f652e0]
  8:
/usr/lib64/ceph/libceph-common.so.2(+0x4442e0) [0x7f9a23f652e0]
  9:
(MonClient::_add_conns()+0x242) [0x7f9a23f5fa42]
  10:
(MonClient::_reopen_session(int)+0x428) [0x7f9a23f60518]
  11:
(Mgr::init()+0x384)
[0x5604667a6434]
  12:
/usr/bin/ceph-mgr(+0x1af271) [0x5604667ae271]
  13:
/usr/bin/ceph-mgr(+0x11364d) [0x56046671264d]
  14:
(Finisher::finisher_thread_entry()+0x175) [0x7f9a23d10645]
  15:
/lib64/libc.so.6(+0x9f802) [0x7f9a23698802]
  16:
/lib64/libc.so.6(+0x3f450) [0x7f9a23638450]
  NOTE: a copy of the
executable, or `objdump -rdS ` is needed to interpret this.

I have the impression that the MGRs are ejected by the monitors, however
after debugging monitor, I don't see anything abnormal on the monitor
side
(if I haven't missed something).

All we can see is that we get an exception on the "_add_conn" method (
https://github.com/ceph/ceph/blob/v17.2.6/src/mon/MonClient.cc #L775)

Version : 17.2.6-170.el9cp (RHCS6)




___

ceph-users mailing list -- ceph-users@ce

[ceph-users] Which RHEL/Fusion/CentOS/Rocky Package Contains cephfs-shell?

[duluxoz]

Hi All,

The subject pretty much says it all: I need to use cephfs-shell and its 
not installed on my Ceph Node, and I can't seem to locate which package 
contains it - help please.  :-)


Cheers

Dulux-Oz
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io