[ceph-users] Re: Single ceph client usage with multiple ceph cluster

[Mosharaf Hossain]

Hello Markus
Thank you for your direction.
I would like to let you know that the way you show it is quite meaningful
but I am afraid how the ceph system would identify the configuration file
as by default it uses ceph. conf in /etc/ceph folder. Can we define the
config file as we want?

It will be helpful to give or show us guidelines to add the ceph client to
multiple clusters.




Regards
Mosharaf Hossain
Deputy Manager, Product Development
IT Division

Bangladesh Export Import Company Ltd.

Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh

Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757

Cell: +8801787680828, Email: mosharaf.hoss...@bol-online.com, Web:
www.bol-online.com




On Tue, Nov 2, 2021 at 7:01 PM Markus Baier <
markus.ba...@bcs.tu-darmstadt.de> wrote:

> Hello,
>
> yes you can use a single server to operate multiple clusters.
> I have a configuration running, with two independent ceph clusters
> running on the same node (of course multiple nodes for the two clusters)
>
> The trick is to work with multiple ceph.conf files, I use two
> seperate ceph.conf files under /etc/ceph/
> One is called .conf and the other .conf
>
> Every cluster uses it's seperate network interfaces, so I use four 10GbE
> Interfaces
> for the two clusters, but you can also use vlans togehter with a 100GbE
> Interface or a 100GbE NIC that can provide virtual ports for the separation
> of the networks and the distribution of the network load.
>
> Every cluster uses also a seperate keyring, so e.g for the first
> cluster you have a keyring named .mon.keyring
> and for the second one .mon.keyring
> inside of the /etc/ceph folder.
>
> To administrate the whole thing, ceph provide
> the --cluster parameter for the command line programs.
> So ceph --cluster  -s
> will show the outputs for cluster one and
> ceph --cluster  -s
> for the cluster two
>
>
> Regards
> Markus Baier
>
> Am 02.11.21 um 13:30 schrieb Mosharaf Hossain:
> > Hi Users
> > We have two ceph clusters in our lab. We are experimenting to use a
> single
> > server as a client for two ceph clusters. Can we use the same client
> server
> > to store keyring for different clusters in ceph.conf file.
> >
> >
> >
> >
> > Regards
> > Mosharaf Hossain
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Need urgent help for ceph health error issue

[Md. Hejbul Tawhid MUNNA]

Hi,

Yes, we have added new osd. Previously we had only one type disk, hdd. now
we have added ssd disk separate them with replicated_rule and device class

ID CLASS WEIGHT  REWEIGHT SIZEUSE AVAIL   %USE  VAR  PGS
 0   hdd 5.57100  1.0 5.6 TiB 1.8 TiB 3.8 TiB 31.61 1.04  850
 1   hdd 5.57100  1.0 5.6 TiB 1.6 TiB 4.0 TiB 29.07 0.96  830
 2   hdd 5.57100  1.0 5.6 TiB 1.6 TiB 4.0 TiB 27.98 0.92  820
 3   hdd 5.57100  1.0 5.6 TiB 1.3 TiB 4.2 TiB 23.74 0.78  696
 4   hdd 5.57100  1.0 5.6 TiB 1.8 TiB 3.8 TiB 32.15 1.06  866
 5   hdd 5.57100  1.0 5.6 TiB 1.9 TiB 3.7 TiB 34.38 1.13  835
 6   hdd 5.57100  1.0 5.6 TiB 1.4 TiB 4.2 TiB 25.10 0.83  702
 7   hdd 5.57100  1.0 5.6 TiB 1.5 TiB 4.1 TiB 26.34 0.87  680
 8   hdd 5.57100  1.0 5.6 TiB 1.6 TiB 3.9 TiB 29.25 0.96  811
 9   hdd 5.57100  1.0 5.6 TiB 2.0 TiB 3.6 TiB 36.01 1.18  819
10   hdd 5.57100  1.0 5.6 TiB 1.5 TiB 4.0 TiB 27.66 0.91  848
11   hdd 5.57100  1.0 5.6 TiB 1.5 TiB 4.0 TiB 27.63 0.91  844
12   hdd 5.57100  1.0 5.6 TiB 1.5 TiB 4.0 TiB 27.68 0.91  719
13   hdd 5.57100  1.0 5.6 TiB 2.0 TiB 3.6 TiB 35.08 1.15  832
14   hdd 5.57100  1.0 5.6 TiB 1.8 TiB 3.8 TiB 32.09 1.06  840
15   hdd 5.57100  1.0 5.6 TiB 1.8 TiB 3.8 TiB 32.53 1.07  825
16   hdd 5.57100  1.0 5.6 TiB 1.4 TiB 4.2 TiB 24.85 0.82  855
17   hdd 5.57100  1.0 5.6 TiB 2.1 TiB 3.5 TiB 37.46 1.23  899
18   hdd 5.57100  1.0 5.6 TiB 1.7 TiB 3.9 TiB 30.03 0.99  856
19   hdd 5.57100  1.0 5.6 TiB 1.6 TiB 4.0 TiB 28.20 0.93  850
20   hdd 5.57100  1.0 5.6 TiB 1.7 TiB 3.8 TiB 31.38 1.03  776
21   hdd 5.57100  1.0 5.6 TiB 1.5 TiB 4.0 TiB 27.56 0.91  868
22   hdd 5.57100  1.0 5.6 TiB 1.9 TiB 3.6 TiB 34.61 1.14  836
23   hdd 5.57100  1.0 5.6 TiB 1.8 TiB 3.8 TiB 32.30 1.06  849
24   hdd 5.57100  1.0 5.6 TiB 1.7 TiB 3.9 TiB 30.11 0.99  847
25   hdd 5.57100  1.0 5.6 TiB 1.9 TiB 3.6 TiB 34.90 1.15  887
26   hdd 5.57100  1.0 5.6 TiB 1.6 TiB 3.9 TiB 29.53 0.97  792
27   hdd 5.57100  1.0 5.6 TiB 2.2 TiB 3.4 TiB 38.98 1.28  878
28   hdd 5.57100  1.0 5.6 TiB 1.8 TiB 3.8 TiB 32.53 1.07  845
29   hdd 5.57100  1.0 5.6 TiB 1.8 TiB 3.7 TiB 32.95 1.08  853
30   hdd 5.57100  1.0 5.6 TiB 1.7 TiB 3.9 TiB 30.42 1.00  838
31   hdd 5.57100  1.0 5.6 TiB 1.5 TiB 4.0 TiB 27.39 0.90  823
32   hdd 5.57100  1.0 5.6 TiB 1.7 TiB 3.9 TiB 30.57 1.01  827
33   hdd 5.57100  1.0 5.6 TiB 1.8 TiB 3.8 TiB 32.18 1.06  860
34   hdd 5.57100  1.0 5.6 TiB 1.7 TiB 3.9 TiB 30.70 1.01  821
35   hdd 5.57100  1.0 5.6 TiB 1.7 TiB 3.9 TiB 29.99 0.99  897
36   hdd 5.57100  1.0 5.6 TiB 1.4 TiB 4.1 TiB 25.54 0.84  828
37   hdd 5.57100  1.0 5.6 TiB 1.7 TiB 3.9 TiB 30.09 0.99  784
38   hdd 5.57100  1.0 5.6 TiB 1.8 TiB 3.8 TiB 31.92 1.05  834
39   hdd 5.57100  1.0 5.6 TiB 2.2 TiB 3.4 TiB 39.44 1.30  887
40   ssd 1.81898  1.0 1.8 TiB 141 GiB 1.7 TiB  7.55 0.25 1035
41   ssd 1.81898  1.0 1.8 TiB  94 GiB 1.7 TiB  5.07 0.17 1043
TOTAL 226 TiB  69 TiB 158 TiB 30.40
MIN/MAX VAR: 0.17/1.30  STDDEV: 6.38


# ceph health detail
HEALTH_ERR 11759382/17719047 objects misplaced (66.366%); Degraded data
redundancy: 1127230/17719047 objects degraded (6.362%), 644 pgs degraded,
652 pgs undersized; Degraded data redundancy (low space): 161 pgs
backfill_toofull
OBJECT_MISPLACED 11759382/17719047 objects misplaced (66.366%)
PG_DEGRADED Degraded data redundancy: 1127230/17719047 objects degraded
(6.362%), 644 pgs degraded, 652 pgs undersized
pg 16.30c is stuck undersized for 40925.003419, current state
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, last
acting [8,24]
pg 17.2dd is active+undersized+degraded+remapped+backfill_wait, acting
[27,34]
pg 17.2df is stuck undersized for 29653.050654, current state
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, last
acting [20,31]
pg 17.2e1 is stuck undersized for 40852.659824, current state
active+undersized+degraded+remapped+backfill_wait, last acting [19,32]
pg 17.2e5 is stuck undersized for 40947.285931, current state
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, last
acting [2,14]
pg 17.2e6 is stuck undersized for 40853.656156, current state
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, last
acting [15,27]
pg 17.2eb is stuck undersized for 29653.065202, current state
active+undersized+degraded+remapped+backfill_wait, last acting [39,26]
pg 17.2f0 is stuck undersized for 40951.126668, current state
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, last
acting [13,27]
pg 17.2f2 is stuck undersized for 40931.707178, current state
active+undersized+degraded+remapped+backfill_wait, last acting [15,28]
pg 17.2f7 is stuck undersized for 29653.031067, current state
active+undersized+degraded+remapped+backfill_wait, last acting [23,28]
pg 17.2fb is stuck undersized for 40853.695152, current state

[ceph-users] v16.2.6 PG peering indefinitely after cluster power outage

[Eric Alba]

I've been trying to get ceph to force the PG to a good state but it
continues to give me a single PG peering. This is a rook-ceph cluster on
VMs (hosts went out for a brief period) and I can't figure out how to get
this 1GB or so of data to become available to the client. This occurred
during a cluster expansion. This gave the added joy of clock skew. Another
3 disks were being added to extend the cluster storage. This is a 3 node
cluster with 6 200GB Disks each node.

  cluster:
id: 31689324-f5ba-4aa4-8244-aa09a1119dc3
health: HEALTH_OK

  services:
mon: 3 daemons, quorum a,c,d (age 41m)
mgr: a(active, since 2h)
osd: 18 osds: 18 up (since 17m), 18 in (since 2h); 1 remapped pgs

  data:
pools:   2 pools, 768 pgs
objects: 184.38k objects, 707 GiB
usage:   1.4 TiB used, 2.1 TiB / 3.5 TiB avail
pgs: 0.130% pgs not active
 767 active+clean
 1   peering

  io:
client:   14 KiB/s wr, 0 op/s rd, 1 op/s wr

  progress:
Global Recovery Event (2h)
  [===.] (remaining: 9s)

PG map for problematic PG

[root@rook-ceph-tools-5b54fb98c-kjt5t /]# ceph pg map 2.11c
osdmap e11328 pg 2.11c (2.11c) -> up [17,12] acting [4]

Don't see OSD 4 in pg dump:
[root@rook-ceph-tools-5b54fb98c-kjt5t /]# ceph pg dump 2> /dev/null | egrep
'2\.11c|MISS'
PG_STAT  OBJECTS  MISSING_ON_PRIMARY  DEGRADED  MISPLACED  UNFOUND  BYTES
OMAP_BYTES*  OMAP_KEYS*  LOG   DISK_LOG  STATE STATE_STAMP
 VERSION REPORTEDUP  UP_PRIMARY
 ACTING  ACTING_PRIMARY  LAST_SCRUB SCN
2.11c249   0 0  00
 10237583360   0  5352  5352   peering
 2021-12-08T22:44:36.691204+3669'8643974  11329:7414
[17,12]  17 [17,12]  17   3162'8635517  200



ceph pg 2.11c query
{
"snap_trimq": "[]",
"snap_trimq_len": 0,
"state": "unknown",
"epoch": 12191,
"up": [
17,
12
],
"acting": [
17,
12
],
"info": {
"pgid": "2.11c",
"last_update": "3669'8643974",
"last_complete": "3669'8643974",
"log_tail": "3174'8638622",
"last_user_version": 8643974,
"last_backfill":
"2:38e2dedc:::rbd_data.1d091ec6540582.158e:head",
"purged_snaps": [],
"history": {
"epoch_created": 3286,
"epoch_pool_created": 13,
"last_epoch_started": 3725,
"last_interval_started": 3724,
"last_epoch_clean": 3544,
"last_interval_clean": 3543,
"last_epoch_split": 3286,
"last_epoch_marked_full": 0,
"same_up_since": 9923,
"same_interval_since": 12191,
"same_primary_since": 12191,
"last_scrub": "3162'8635517",
"last_scrub_stamp": "2021-12-06T22:16:26.082945+",
"last_deep_scrub": "3122'8608768",
"last_deep_scrub_stamp": "2021-12-04T02:35:48.882906+",
"last_clean_scrub_stamp": "2021-12-06T22:16:26.082945+",
"prior_readable_until_ub": 0
},
"stats": {
"version": "3669'8643974",
"reported_seq": 8277,
"reported_epoch": 12191,
"state": "peering",
"last_fresh": "2021-12-08T22:59:01.450813+",
"last_change": "2021-12-08T22:59:01.450813+",
"last_active": "0.00",
"last_peered": "0.00",
"last_clean": "0.00",
"last_became_active": "0.00",
"last_became_peered": "0.00",
"last_unstale": "2021-12-08T22:59:01.450813+",
"last_undegraded": "2021-12-08T22:59:01.450813+",
"last_fullsized": "2021-12-08T22:59:01.450813+",
"mapping_epoch": 12191,
"log_start": "3174'8638622",
"ondisk_log_start": "3174'8638622",
"created": 3286,
"last_epoch_clean": 3544,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "3162'8635517",
"last_scrub_stamp": "2021-12-06T22:16:26.082945+",
"last_deep_scrub": "3122'8608768",
"last_deep_scrub_stamp": "2021-12-04T02:35:48.882906+",
"last_clean_scrub_stamp": "2021-12-06T22:16:26.082945+",
"log_size": 5352,
"ondisk_log_size": 5352,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 1023758336,
"num_objects": 249,
"num_object_clones": 0,

[ceph-users] Re: 50% IOPS performance drop after upgrade from Nautilus 14.2.22 to Octopus 15.2.15

[Marc]

I am still on nautilus, albeit a tiny cluster. I would not mind doing some 
tests for comparison if necessary.


> 
> Hi Frank, thanks for the input. Im still a bit sceptical to be honest
> that this is all, since a.) our bench values are pretty stable over time
> (natilus times and octopus times) with a variance of maybe 20% which i
> would put on normal cluster load.
> 
> Furthermore the HDD pool also halved its performance and the IO
> waitstates also halved and the raw OSD IO Utilisation dropped by 50%
> since the update.
> 
> From old testings (actually done with FIO) i still see, that in our old
> setup (10GE only) we could achieve 310k IOPS on NVME only test storage
> and our current SSD’s do around 35k per Disk, so i guess we should be
> able to reach higher values than we do right now with enough clients.
> 
> i need to know if there is a proper explanation for the waitstates vs.
> performance drop… ;-)
> 
> Cheers Kai
> 
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Migration from CentOS7/Nautilus to CentOS Stream/Pacific

[Edward R Huyer]

> -Original Message-
> From: Carlos Mogas da Silva 
> Sent: Wednesday, December 8, 2021 1:26 PM
> To: Edward R Huyer ; Marc ;
> ceph-users@ceph.io
> Subject: Re: [ceph-users] Re: Migration from CentOS7/Nautilus to CentOS
> Stream/Pacific
> 
> On Wed, 2021-12-08 at 16:42 +, Edward R Huyer wrote:
> >
> > I did a similar upgrade over the summer, only from RHEL7 to RHEL8, and
> > the "backup key directories/files, clean install, restore directories"
> > is essentially what you have to do, with the extra step of telling
> > Ceph to re-detect your OSDs and recreate the startup scripts.  That
> > last bit is easier if all your OSDs are already LVM-based rather than
> simple.  If you have simple OSDs, make sure you grab the JSON files for them
> in /etc/ceph.
> >
> > I can try to figure out what commands I used for some of this if needed.
> 
> 
> If that's cool with you, I wouldn't mind since it can help a lot just to 
> figure out
> what exactly needs to be done (you can send it of-list if you prefer). OSDs
> are already LVM based.

I'll put it here in case it's useful to someone else in the future.

I'll note that I'm reconstructing something from months ago that I was figuring 
out as I went, expecting to never need it again.  Some things may be incomplete 
or unnecessary.  Also, this was RHEL7->8, not Cent.  Differences should be 
minimal, but YMMV.
 
Ok, you're going to want to preserve the following:
/etc/ceph
/etc/systemd/system/ceph-mgr.target.wants
/etc/systemd/system/ceph-mon.target.wants  (These should let you start the mons 
and mgrs without going through the process of creating new ones.)
/var/lib/ceph/bootstrap-mds
/var/lib/ceph/bootstrap-mgr
/var/lib/ceph/bootstrap-osd
/var/lib/ceph/bootstrap-rbd
/var/lib/ceph/bootstrap-rbd-mirror
/var/lib/ceph/bootstrap-rgw
/var/lib/ceph/mgr
/var/lib/ceph/mon

Note that the /var/lib/ceph directories do not include /var/lib/ceph/osd.  All 
the data in the directories under there lives on the OSDs themselves, so you 
don't need to carry that over.  /var/lib/ceph/mon is the most important, since 
that's where your mon's brains live.  Anyway, tar all that up and copy the 
tarball off the server.

Do your whole reinstall of the OS and Ceph.  Don't touch your OSD drives, 
obviously.  Copy your tarball over and extract the contents to where they need 
to go.  Open your firewall, if necessary.  If Cent uses firewalld, this should 
do it:
firewall-cmd --add-service ceph
firewall-cmd --add-service ceph-mon
firewall-cmd --permanent --add-service ceph
firewall-cmd --permanent --add-service ceph-mon

Adjust as needed for your environment.

Start the monitor with
systemctl start ceph-mon.target
Make sure that's running correctly before continuing.  I think this also 
autostarts the mgr.

Tell ceph-volume to find all the existing lvm-based OSDs, create the relevant 
startup scripts, and start them.
ceph-volume lvm activate --all 

Then wait for recovery.  That should be it as far as the reinstall goes.  If 
you do in fact have some simple OSDs lying around after all, you'll need
ceph-volume simple activate --file /etc/ceph/osd/[filename]

Hope this helps.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Need urgent help for ceph health error issue

[Nathan Fish]

You should probably stop all client mounts to avoid any more writes,
temporarily raise full ratios just enough to get it online, then
delete something. Never let it get this full.

On Wed, Dec 8, 2021 at 1:27 PM Md. Hejbul Tawhid MUNNA
 wrote:
>
> Hi,
>
> Overall status: HEALTH_ERR
> PG_DEGRADED_FULL: Degraded data redundancy (low space): 19 pgs
> backfill_toofull
> OBJECT_MISPLACED: 12359314/17705640 objects misplaced (69.804%)
> PG_DEGRADED: Degraded data redundancy: 1707105/17705640 objects degraded
> (9.642%), 1979 pgs degraded, 1155 pgs undersized
>
> Any advice to resolve the issue. Its running in production
>
> Please let me know for any further information.
>
> Regards,
> Munna
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Need urgent help for ceph health error issue

[Prayank Saxena]

Hi Munna,

Have you added osd’s in the cluster recently?
If yes, i think you have to re-weight the osd’s which you have added to
lower values and slowly increase the weight one by one.

Also, please share output of ‘ceph osd df’ and ‘ceph health details’

On Wed, 8 Dec 2021 at 11:56 PM, Md. Hejbul Tawhid MUNNA 
wrote:

> Hi,
>
> Overall status: HEALTH_ERR
> PG_DEGRADED_FULL: Degraded data redundancy (low space): 19 pgs
> backfill_toofull
> OBJECT_MISPLACED: 12359314/17705640 objects misplaced (69.804%)
> PG_DEGRADED: Degraded data redundancy: 1707105/17705640 objects degraded
> (9.642%), 1979 pgs degraded, 1155 pgs undersized
>
> Any advice to resolve the issue. Its running in production
>
> Please let me know for any further information.
>
> Regards,
> Munna
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
-- 




Regards
Prayank Saxena
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Need urgent help for ceph health error issue

[Marc]

> 
> Overall status: HEALTH_ERR
> PG_DEGRADED_FULL: Degraded data redundancy (low space): 19 pgs
> backfill_toofull
> OBJECT_MISPLACED: 12359314/17705640 objects misplaced (69.804%)
> PG_DEGRADED: Degraded data redundancy: 1707105/17705640 objects degraded
> (9.642%), 1979 pgs degraded, 1155 pgs undersized
> 
> Any advice to resolve the issue. Its running in production
> 
> Please let me know for any further information.
> 


Maybe play with these? 

ceph tell osd.* injectargs '--mon_osd_full_ratio=0.97'
ceph tell osd.* injectargs '--mon_osd_backfillfull_ratio=0.95'

ceph tell osd.* injectargs '--mon_osd_full_ratio=0.95'
ceph tell osd.* injectargs '--mon_osd_backfillfull_ratio=0.90'
 
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Need urgent help for ceph health error issue

[Md. Hejbul Tawhid MUNNA]

Hi,

Overall status: HEALTH_ERR
PG_DEGRADED_FULL: Degraded data redundancy (low space): 19 pgs
backfill_toofull
OBJECT_MISPLACED: 12359314/17705640 objects misplaced (69.804%)
PG_DEGRADED: Degraded data redundancy: 1707105/17705640 objects degraded
(9.642%), 1979 pgs degraded, 1155 pgs undersized

Any advice to resolve the issue. Its running in production

Please let me know for any further information.

Regards,
Munna
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: v16.2.7 Pacific released

[Robert Sander]

Am 08.12.21 um 19:17 schrieb Robert Sander:

The NFS topic has not even been mentioned in the release announcement 
email


This is obviously not true, I just read over that paragraph. Blame on me.

Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin

http://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Zwangsangaben lt. §35a GmbHG:
HRB 220009 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: v16.2.7 Pacific released

[Robert Sander]

Am 08.12.21 um 17:57 schrieb Sebastian Wagner:


Unfortunately this wasn't the case and I agree this is not great.


What bothers me most is the following:

With cephadm, the orchestrator and containers the Ceph project aims to 
make it easy for admins to operate a cluster. This goal has been 
acknowledged by most of us, I think.


Admins are now teached that an upgrade is as easy as running "ceph orch 
upgrade start --version 16.2.7".


And now something like this comes along and after the orchestrator has 
been finished with the upgrade there is no NFS service any more.


This is the opposite of an easy adminstration.

The NFS topic has not even been mentioned in the release announcement 
email and there is no 16.2.7 version mentioned on 
https://docs.ceph.com/en/pacific/releases/ or 
https://docs.ceph.com/en/latest/releases/.


How would the average admin know what to do?

I was lucky as this was just happening on a test cluster.

Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin

http://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Zwangsangaben lt. §35a GmbHG:
HRB 220009 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: v16.2.7 Pacific released

[Robert Sander]

Hi,

Am 08.12.21 um 17:57 schrieb Sebastian Wagner:


In any case, here are the manual steps that are performed by the
migration automatically, in case something goes wrong:

https://github.com/ceph/ceph/pull/44252


There was no automatic migration of the old NFS exports to the new 
default pool ".nfs".


Why has the feature to configure a specific cephx key been removed?
What key is now used by nfs-ganesha to access the CephFS?

Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin

http://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Zwangsangaben lt. §35a GmbHG:
HRB 220009 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] CephFS Metadata Pool bandwidth usage

[Andras Sali]

Hi All,

We have been observing that if we let our MDS run for some time, the
bandwidth usage of the disks in the metadata pool starts increasing
significantly (whilst IOPS is about constant), even though the number of
clients, the workloads or anything else doesn't change.

However, after restarting the MDS, the issue goes away for some time and
the same workloads require 1/10th of the metadata disk bandwidth whilst
doing the same IOPS.

We run our CephFS cluster in a cloud environment where the disk throughput
/ bandwidth capacity is quite expensive to increase and we are hitting
bandwidth / throughput limits, even though we still have a lot of IOPS
capacity left.

We suspect that somehow the journaling of the MDS becomes more extensive
(i.e. larger journal updates for each operation), but we couldn't really
pin down which parameter might affect this.

I attach a plot of how the Bytes / Operation (throughput in MBps / IOPS)
evolves over time, when we restart the MDS, it drops to around 32kb (even
though the min block size for the metadata pool OSDs is 4kb in our
settings) and then increases over time to around 300kb.

Any ideas on how to "fix" this and have a significantly lower bandwidth
usage would be really-really appreciated!

Thank you and kind regards,

Andras
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: v16.2.7 Pacific released

[Sebastian Wagner]

Hi Robert,

it would have been much better to avoid this NFS situation altogether by
avoiding those different implementations in the first place.
Unfortunately this wasn't the case and I agree this is not great.

In any case, here are the manual steps that are performed by the
migration automatically, in case something goes wrong:

https://github.com/ceph/ceph/pull/44252

I hope that helps!

Best,
Sebastian


Am 08.12.21 um 10:42 schrieb Robert Sander:
> Am 08.12.21 um 01:11 schrieb David Galloway:
>
>> * Cephadm & Ceph Dashboard: NFS management has been completely reworked
>> to ensure that NFS exports are managed consistently across the different
>> Ceph components. Prior to this, there were 3 incompatible
>> implementations for configuring the NFS exports: Ceph-Ansible/OpenStack
>> Manila, Ceph Dashboard and 'mgr/nfs' module. With this release the
>> 'mgr/nfs' way becomes the official interface, and the remaining
>> components (Cephadm and Ceph Dashboard) adhere to it. While this might
>> require manually migrating from the deprecated implementations, it will
>> simplify the user experience for those heavily relying on NFS exports.
>
> This change is introduced in a point release?
>
> After upgrading a cluster all NFS shares have to be configured again
> and in the meantime NFS services do not work. Not so great IMHO.
>
> Regards
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Migration from CentOS7/Nautilus to CentOS Stream/Pacific

[Edward R Huyer]

> On Wed, 2021-12-08 at 16:06 +, Marc wrote:
> > >
> > > It isn't possible to upgrade from CentOS 7 to anything... At least
> > > without required massive hacks that may of may not work (and most
> > > likely won't).
> >
> > I meant wipe the os disk, install whatever, install nautilus and put
> > back some dirs from the previous os, like /etc/ceph and
> > /var/lib/ceph/. Get it to work for one node, and you have your blue print
> for the rest. I was planning to do it like this in the near future.
> >
> 
> Was afraid this wasn't possible due to total different architecture (Nautilus
> isn't docker/podman based yet). Still, my cluster only has 2 nodes with all 
> the
> data on one of it, basically I'm running a RAID1 on steroids.

You don't *have* to run Pacific using Docker/Podman.

I did a similar upgrade over the summer, only from RHEL7 to RHEL8, and the 
"backup key directories/files, clean install, restore directories" is 
essentially what you have to do, with the extra step of telling Ceph to 
re-detect your OSDs and recreate the startup scripts.  That last bit is easier 
if all your OSDs are already LVM-based rather than simple.  If you have simple 
OSDs, make sure you grab the JSON files for them in /etc/ceph.

I can try to figure out what commands I used for some of this if needed.

Once the operating systems are reinstalled and the cluster is functioning 
again, then you upgrade from Nautilus to Octopus, wait for the Octopus OSD 
format conversion to happen, then upgrade to Pacific, *then* worry about 
whether you want to switch to Docker/Podman.

Oh, and a word of warning:  Pacific only supports cephx v2 authentication.  If 
you have clients doing kernel RBD mounting, make sure their kernels are at 
least 4.9.150, 4.14.86, 4.19, or later.  Prior to those versions, the kernel 
driver doesn't support cephx v2, and you'll have a Bad Time.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Migration from CentOS7/Nautilus to CentOS Stream/Pacific

[Jan Kasprzak]

Carlos,

Carlos Mogas da Silva wrote:
: >From what I can gather, this will not be smooth at all, since I can't make 
an inplace upgrade of the
: OS first and then Ceph and neither other way around. So the idea is to create 
a total new Ceph
: cluster from scratch and migrate the data from one to another. The question 
is, how do I do this
: last step?

I've just finished the in-place migration from C7 to C8Stream myself.
What I did was:

- upgrade the cluster to C7 + Octopus, because Octopus packages are available
  both for C7 and C8.

- upgrade the OS on the mons one by one, preserving the
  /var/lib/ceph/mon/ceph-* directories (backup before and restore
  afterwards is OK). My mon hosts also run an instance of mgr and rgw,
  so start also these. I did this at least a month ago, so I don't exactly
  remember whether I did some more magic somewhere :-). But the cluster has
  been running happily for at least a month with C8Stream+Octopus mons
  and C7+Octopus OSDs.

- ceph osd set noout

- upgrade the osd hosts one by one:

  - create a kickstart.cfg file which deletes all the non-ceph partitions,
creates new ones, and installs C8Stream there.
I have OSD data on whole disks without partitions, so the installer does
not have a chance to overwrite them accidentally - it would fail when
trying to delete the old system partitions first, even if the order of
the disks gets changed somehow. Something like this:

ignoredisk --only-use=nvme0n1,nvme1n1
clearpart --list=nvme0n1p1,nvme0n1p2,nvme0n1p3,nvme1n1p1,nvme1n1p2,nvme1n1p3

  - I set up Kickstart.cfg as a minimal install of C8Stream with ansible SSH
public key added to /root/.ssh/authorized_keys in %post (remember
to restorecon -R /root/.ssh in %post as well).

  - I booted the installer using PXE+UEFI with GRUB, so that I can include the
ks=.../kickstart.cfg option there and don't have to enter it manually.
Just don't make the grub entry with disk-overwriting kickstart.cfg the
default one :-)

- In dhcpd.conf, I have

  next-server my.tftp.server;
  filename "c8stream/grubx64.efi";

- I've put grubx64.efi, vmlinuz, grub.cfg and initrd.img to
  my.tftp.server:/tftpboot

- the grub.cfg entries are:

  set timeout=120
  default skip

  menuentry 'Exit GRUB (and try next boot option)' --id skip {
  echo Bye :]
  exit
  }
  menuentry 'CentOS 8 Stream Kickstart (REWRITES DISKS!!!)' {
  linuxefi /c8stream/vmlinuz ip=dhcp 
inst.ks=http://my.http.server/kicksart.cfg
  initrdefi /c8stream/initrd.img
  }

  - Then I used ansible to configure the rest (Ceph repository, packages,
and some other tools we need - snmpd, ...).

  - Make sure to have /var/lib/ceph/bootstrap-osd/ceph.keyring installed
with the correct bootstrap-osd key.

  - after the osd host is up and running, re-attach the OSD volumes with
"ceph-volume lvm activate --all"

  - after "ceph -s" reports no other problem than the noout flag being set,
continue with the next OSD host.

- ceph osd unset noout

- optionally, upgrade to Pacific (I did not do that part yet)

For me, it still was a lot of manual work, because I also did a BIOS upgrade
on each OSD host, so I had to reconfigure the BIOS NVRAM as well. I guess
it took me about 30 to 60 minutes per host.

Hope this helps,

-Yenya

-- 
| Jan "Yenya" Kasprzak  |
| http://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 |
We all agree on the necessity of compromise. We just can't agree on
when it's necessary to compromise. --Larry Wall
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Migration from CentOS7/Nautilus to CentOS Stream/Pacific

[Carlos Mogas da Silva]

(replying to list again as I forgot to "group reply")

On Wed, 2021-12-08 at 16:06 +, Marc wrote:
> > 
> > It isn't possible to upgrade from CentOS 7 to anything... At least
> > without required massive hacks
> > that may of may not work (and most likely won't).
> 
> I meant wipe the os disk, install whatever, install nautilus and put back 
> some dirs from the
> previous os, like /etc/ceph and /var/lib/ceph/. Get it to work for one node, 
> and you have your
> blue print for the rest. I was planning to do it like this in the near future.
> 

Was afraid this wasn't possible due to total different architecture (Nautilus 
isn't docker/podman
based yet). Still, my cluster only has 2 nodes with all the data on one of it, 
basically I'm running
a RAID1 on steroids.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Migration from CentOS7/Nautilus to CentOS Stream/Pacific

[Marc]

> 
> From what I can gather, this will not be smooth at all, since I can't
> make an inplace upgrade of the
> OS first and then Ceph and neither other way around. 

I think easier would be to upgrade one node at a time from centos7 to ... + 
nautilus. And when that is done do the upgrade to pacific. 
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Migration from CentOS7/Nautilus to CentOS Stream/Pacific

[Carlos Mogas da Silva]

Hey guys.

>From what I can gather, this will not be smooth at all, since I can't make an 
>inplace upgrade of the
OS first and then Ceph and neither other way around. So the idea is to create a 
total new Ceph
cluster from scratch and migrate the data from one to another. The question is, 
how do I do this
last step?

Thanks in advance
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RBDMAP clients rendering theirselfs as "Jewel" in "Luminous" ceph cluster

[Alex Gorbachev]

You may be running into a bug with the way client features are
interpreted.  You may want to review these links, below.  Also, you can
check your running kernel with e.g. uname -a.

https://www.mail-archive.com/ceph-users@ceph.io/msg03365.html

https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RUBXOY2L4JD7AYXHTTTXNJI4BPE6S7TX/

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-May/027002.html
--
Alex Gorbachev
https://alextelescope.blogspot.com



On Wed, Dec 8, 2021 at 8:38 AM Kamil Kuramshin 
wrote:

> I have already tried kernel 4.19  - newest available from
> stretch-backports repo - nothing changed.
> Any more suggestions? Maybe I do not know about some kind of configuration
> parameters to make linux kernel 4.19 to not use ceph features related to
> Jewel release?
>
>
> > One of:
> >
> > client
> > version
> > kernel-mainline
> > 4.13
> > kernel-el 3.10.0-862
> > librados  12.2.0
> >
> >
> > k
> >
> >
> > > On 8 Dec 2021, at 16:05, Kamil Kuramshin 
> wrote:
> > >
> > > What kernel version should be used to get rid of "Jewel" version
> features from sessions list?
> >
> >
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RBDMAP clients rendering theirselfs as "Jewel" in "Luminous" ceph cluster

[Konstantin Shalygin]

Obviously part of your clients may be upgraded, part is not. Or was upgraded, 
but not rebooted
Double check client IP's fro mon sessions


k

> On 8 Dec 2021, at 16:38, Kamil Kuramshin  wrote:
> 
> I have already tried kernel 4.19  - newest available from stretch-backports 
> repo - nothing changed. 
> Any more suggestions? Maybe I do not know about some kind of configuration 
> parameters to make linux kernel 4.19 to not use ceph features related to 
> Jewel release?

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RBDMAP clients rendering theirselfs as "Jewel" in "Luminous" ceph cluster

[Konstantin Shalygin]

One of:

client
version
kernel-mainline 
4.13
kernel-el   3.10.0-862
librados12.2.0


k


> On 8 Dec 2021, at 16:05, Kamil Kuramshin  wrote:
> 
> What kernel version should be used to get rid of "Jewel" version features 
> from sessions list?

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: CEPHADM_STRAY_DAEMON with iSCSI service

[Paul Giralt (pgiralt)]

https://tracker.ceph.com/issues/5

-Paul

Sent from my iPhone

On Dec 8, 2021, at 8:00 AM, Robert Sander  wrote:

Hi,

i just upgraded to 16.2.7 and deployed an iSCSI service.

Now I get for each configured target three stray daemons
(tcmu-runner) that are not managed by cephadm:

HEALTH_WARN 6 stray daemon(s) not managed by cephadm
[WRN] CEPHADM_STRAY_DAEMON: 6 stray daemon(s) not managed by cephadm
   stray daemon tcmu-runner.cephtest20:rbd/iscsi01 on host cephtest20 not 
managed by cephadm
   stray daemon tcmu-runner.cephtest20:rbd/iscsi02 on host cephtest20 not 
managed by cephadm
   stray daemon tcmu-runner.cephtest30:rbd/iscsi01 on host cephtest30 not 
managed by cephadm
   stray daemon tcmu-runner.cephtest30:rbd/iscsi02 on host cephtest30 not 
managed by cephadm
   stray daemon tcmu-runner.cephtest40:rbd/iscsi01 on host cephtest40 not 
managed by cephadm
   stray daemon tcmu-runner.cephtest40:rbd/iscsi02 on host cephtest40 not 
managed by cephadm

How can this be resolved?

Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin

http://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Zwangsangaben lt. §35a GmbHG:
HRB 220009 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] CEPHADM_STRAY_DAEMON with iSCSI service

[Robert Sander]

Hi,

i just upgraded to 16.2.7 and deployed an iSCSI service.

Now I get for each configured target three stray daemons
(tcmu-runner) that are not managed by cephadm:

HEALTH_WARN 6 stray daemon(s) not managed by cephadm
[WRN] CEPHADM_STRAY_DAEMON: 6 stray daemon(s) not managed by cephadm
stray daemon tcmu-runner.cephtest20:rbd/iscsi01 on host cephtest20 not 
managed by cephadm
stray daemon tcmu-runner.cephtest20:rbd/iscsi02 on host cephtest20 not 
managed by cephadm
stray daemon tcmu-runner.cephtest30:rbd/iscsi01 on host cephtest30 not 
managed by cephadm
stray daemon tcmu-runner.cephtest30:rbd/iscsi02 on host cephtest30 not 
managed by cephadm
stray daemon tcmu-runner.cephtest40:rbd/iscsi01 on host cephtest40 not 
managed by cephadm
stray daemon tcmu-runner.cephtest40:rbd/iscsi02 on host cephtest40 not 
managed by cephadm

How can this be resolved?

Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin

http://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Zwangsangaben lt. §35a GmbHG:
HRB 220009 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: v16.2.7 Pacific released

[Robert Sander]

Am 08.12.21 um 01:11 schrieb David Galloway:


* Cephadm & Ceph Dashboard: NFS management has been completely reworked
to ensure that NFS exports are managed consistently across the different
Ceph components. Prior to this, there were 3 incompatible
implementations for configuring the NFS exports: Ceph-Ansible/OpenStack
Manila, Ceph Dashboard and 'mgr/nfs' module. With this release the
'mgr/nfs' way becomes the official interface, and the remaining
components (Cephadm and Ceph Dashboard) adhere to it. While this might
require manually migrating from the deprecated implementations, it will
simplify the user experience for those heavily relying on NFS exports.


This change is introduced in a point release?

After upgrading a cluster all NFS shares have to be configured again and 
in the meantime NFS services do not work. Not so great IMHO.


Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin

http://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Zwangsangaben lt. §35a GmbHG:
HRB 220009 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Local NTP servers on monitor node's.

[Anthony D'Atri]

I’ve had good success with this strategy, have the mons chime each other, and 
perhaps have OSD / other nodes against the mons too.  
Chrony >> ntpd
With modern interval backoff / iburst there’s no reason to not have a robust 
set of peers.  

The public NTP pools rotate DNS on some period, so when the quality / jitter 
varies a lot among a given pool you can experience swings.  So depending on the 
scale of one’s organization, it often makes sense to have a set of internal 
stratum 2 servers that servers chime against, which mesh among themselves and 
against both geo-local public servers and a few hand-picked quality *distant* 
servers.  Jitter matters more than latency AIUI.  

Local stratum 1 servers are cool, though getting coax to a DC roof and an 
antenna mounted can be an expensive hassle.  

Success includes a variety of time sources, so that it doesn’t all go to hell 
when some specific server goes weird or disappears, both of which happen.  Eg, 
if there’s a window with sky access, even in an office area, add a couple of 
these (or similar) to the mix, as a source for the workhorse server stratum :

https://www.netburner.com/products/network-time-server/pk70-ex-ntp-network-time-server/#

Not a DC grade item, or a sole solution, but the bang for the buck is 
unbeatable.  


Unless things have changed in the last few years, don’t run NTP servers on VMs. 
 Some network gear can run a server, but be careful with the load it presents 
and how many clients can be supported without impacting the primary roles.  


> On Dec 8, 2021, at 12:14 AM, Janne Johansson  wrote:
> 
> Den ons 8 dec. 2021 kl 02:35 skrev mhnx :
>> I've been building Ceph clusters since 2014 and the most annoying and
>> worst failure is the NTP server faults and having different times on
>> Ceph nodes.
>> 
>> I've fixed few clusters because of the ntp failure.
>> - Sometimes NTP servers can be unavailable,
>> - Sometimes NTP servers can go crazy.
>> - Sometimes NTP servers can respond but systemd-timesyncd can not sync
>> the time without manual help.
>> 
>> I don't want to deal with another ntp problem and because of that I've
>> decided to build internal ntp servers for the cluster.
>> 
>> I'm thinking of creating 3 NTP servers on the 3 monitor nodes to get
>> an internal ntp server cluster.
>> I will use the internal NTP cluster for the OSD nodes and other services.
>> With this way, I believe that I'll always have a stable and fast time server.
> 
> We do something like this. mons gather "calendar time" from outside
> ntp servers, but also peer against eachother, so if/when they drift
> away the mons drift away equal amounts, then all OSDs/RGWs and ceph
> clients pull time from the mons who serve internal ntp based on their
> idea of what time it is.
> 
> Not using systemd, but both chronyd and ntpd allow you to set peers
> for which you sync "sideways" just to keep the pace in-between hosts.
> 
> -- 
> May the most significant bit of your life be positive.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Local NTP servers on monitor node's.

[Janne Johansson]

Den ons 8 dec. 2021 kl 02:35 skrev mhnx :
> I've been building Ceph clusters since 2014 and the most annoying and
> worst failure is the NTP server faults and having different times on
> Ceph nodes.
>
> I've fixed few clusters because of the ntp failure.
> - Sometimes NTP servers can be unavailable,
> - Sometimes NTP servers can go crazy.
> - Sometimes NTP servers can respond but systemd-timesyncd can not sync
> the time without manual help.
>
> I don't want to deal with another ntp problem and because of that I've
> decided to build internal ntp servers for the cluster.
>
> I'm thinking of creating 3 NTP servers on the 3 monitor nodes to get
> an internal ntp server cluster.
> I will use the internal NTP cluster for the OSD nodes and other services.
> With this way, I believe that I'll always have a stable and fast time server.

We do something like this. mons gather "calendar time" from outside
ntp servers, but also peer against eachother, so if/when they drift
away the mons drift away equal amounts, then all OSDs/RGWs and ceph
clients pull time from the mons who serve internal ntp based on their
idea of what time it is.

Not using systemd, but both chronyd and ntpd allow you to set peers
for which you sync "sideways" just to keep the pace in-between hosts.

-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io