An update.
We noticed contradicting output from chrony. "chronyc sources" showed
that chrony was synced. However, we also noted this output:
root@ceph2:/etc/chrony# chronyc activity
200 OK
0 sources online
4 sources offline
0 sources doing burst (return to online)
0 sources doing burst (retur
> @Janne: i will checkout/implement the peer config per your suggestion.
> However what confuses us is that chrony thinks the clocks match, and
> only ceph feels it doesn't. So we are not sure if the peer config will
> actually help in this situation. But time will tell.
Ar ar.
Chrony thinks t
Hi all,
Thanks for all replies!
@Huang: ceph time-sync-status is exactly what I was looking for, thanks!
@Janne: i will checkout/implement the peer config per your suggestion.
However what confuses us is that chrony thinks the clocks match, and
only ceph feels it doesn't. So we are not sure i
If you are just synching to the outside pool, the three hosts may end up
latching on to different outside servers as their definitive sources.
You might want to make one of the three a higher priority source to the
other two and possibly just have it use the outside sources as sync.
Also for
+1 to Janne's suggestion. Also, how many time sources are you using? More
tend to be better and by default chrony has a pretty low limit on the
number of sources if you're using a pool (3 or 4 i think?). You can adjust
it by adding maxsources to the pool line.
pool pool.ntp.org iburst maxsources 8
Den tors 25 apr. 2019 kl 13:00 skrev huang jun :
> mj 于2019年4月25日周四 下午6:34写道：
> >
> > Hi all,
> >
> > On our three-node cluster, we have setup chrony for time sync, and even
> > though chrony reports that it is synced to ntp time, at the same time
> > ceph occasionally reports time skews that can
mj 于2019年4月25日周四 下午6:34写道：
>
> Hi all,
>
> On our three-node cluster, we have setup chrony for time sync, and even
> though chrony reports that it is synced to ntp time, at the same time
> ceph occasionally reports time skews that can last several hours.
>
> See for example:
>
> > root@ceph2:~# ce
Hi all,
On our three-node cluster, we have setup chrony for time sync, and even
though chrony reports that it is synced to ntp time, at the same time
ceph occasionally reports time skews that can last several hours.
See for example:
root@ceph2:~# ceph -v
ceph version 12.2.10 (fc2b1783e3727b
Thanks guys
I was trying these things.
NTP looks well synced.. but drifts away after some hours.
I will soon check if this could be because the BIOS battery of the APU
boxes are empty.
Dominique
--
Your Swiss, Open Source and IPv6 Virtual Machine. Now on
www.datacenterlight.ch
On 8/15/18 10:13
Hi Dominique,
The clock skew warning shows up when your NTP daemon is not synced.
You can see the sync in the output of ntpq -p
This is a synced NTP
# ntpq -p
remote refid st t when poll reach delay offset
jitter
==
syncing from the host server to the VM,
this could be causing the skew as well.
-Brent
-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
Dominque Roux
Sent: Wednesday, August 15, 2018 5:38 AM
To: ceph-users@lists.ceph.com
Subject: [ceph-users
Hi all,
We recently facing clock skews from time to time.
This means that sometimes everything is fine but hours later the warning
appears again.
NTPD is running and configured with the same pool.
Did someone else already had the same issue and could probably help us
to fix this?
Thanks a lot!
Hi Dan,
did you mean "we have not yet..."?
Yes! That's what I meant.
Chrony does much better a job than NTP, at least here :-)
MJ
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> Just to follow-up on this: we have yet experienced a clock skew since we
> starting using chrony. Just three days ago, I know, bit still...
did you mean "we have not yet..."?
> Perhaps you should try it too, and report if it (seems to) work better
> for you as well.
>
> But again, just three
Hi John, list,
On 1-4-2017 16:18, John Petrini wrote:
Just ntp.
Just to follow-up on this: we have yet experienced a clock skew since we
starting using chrony. Just three days ago, I know, bit still...
Perhaps you should try it too, and report if it (seems to) work better
for you as well.
> Op 1 april 2017 om 16:02 schreef John Petrini :
>
>
> Hello,
>
> I'm also curious about the impact of clock drift. We see the same on both
> of our clusters despite trying various NTP servers including our own local
> servers. Ultimately we just ended up adjusting our monitoring to be less
>
Just ntp.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
On 04/01/2017 04:02 PM, John Petrini wrote:
Hello,
I'm also curious about the impact of clock drift. We see the same on
both of our clusters despite trying various NTP servers including our
own local servers. Ultimately we just ended up adjusting our monitoring
to be less sensitive to it since
Hello,
I'm also curious about the impact of clock drift. We see the same on both
of our clusters despite trying various NTP servers including our own local
servers. Ultimately we just ended up adjusting our monitoring to be less
sensitive to it since the clock drift always resolves on its own. Is
Hi,
On 04/01/2017 02:10 PM, Wido den Hollander wrote:
You could try the chrony NTP daemon instead of ntpd and make sure all
MONs are peers from each other.
I understand now what that means. I have set it up according to your
suggestion.
Curious to see how this works out, thanks!
MJ
_
Hi Wido,
On 04/01/2017 02:10 PM, Wido den Hollander wrote:
That warning is there for a reason. I suggest you double-check your NTP and
clocks on the machines. This should never happen in production.
I know... Don't understand why this happens..! Tried both ntpd and
systemd-timesyncd. I did not
> Op 1 april 2017 om 11:17 schreef mj :
>
>
> Hi,
>
> Despite ntp, we keep getting clock skews that auto disappear again after
> a few minutes.
>
> To prevent the unneccerasy HEALTH_WARNs, I have increased the
> mon_clock_drift_allowed to 0.2, as can be seen below:
>
That warning is there
On 04/01/2017 12:49 PM, Wei Jin wrote:
mon_clock_drift_allowed should be used in monitor process, what's the
output of `ceph daemon mon.foo config show | grep clock`?
Ah forgot the actual output:
root@ceph1:~# ceph daemon mon.0 config show | grep clock
"mon_clock_drift_allowed": "0.1",
Hi!
On 04/01/2017 12:49 PM, Wei Jin wrote:
mon_clock_drift_allowed should be used in monitor process, what's the
output of `ceph daemon mon.foo config show | grep clock`?
how did you change the value? command line or config file?
I guess I changed it wrong then... Did it in ceph.conf, like:
On Sat, Apr 1, 2017 at 5:17 PM, mj wrote:
> Hi,
>
> Despite ntp, we keep getting clock skews that auto disappear again after a
> few minutes.
>
> To prevent the unneccerasy HEALTH_WARNs, I have increased the
> mon_clock_drift_allowed to 0.2, as can be seen below:
>
>> root@ceph1:~# ceph --admin-da
Hi,
Despite ntp, we keep getting clock skews that auto disappear again after
a few minutes.
To prevent the unneccerasy HEALTH_WARNs, I have increased the
mon_clock_drift_allowed to 0.2, as can be seen below:
root@ceph1:~# ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show |
gre
Have you tried restarting the mons? Did you change timezone or ran hwclock or
something like that during their lifetime? And if you're running them in
containers, are you providing them with /etc/adjtime and such?
Jan
> On 19 May 2016, at 07:29, Stefan Eriksson wrote:
>
> I'm using hammer and
I'm using hammer and centos 7, and this message wont go away:
health HEALTH_WARN
clock skew detected on mon.ceph01-osd02, mon.ceph01-osd03,
mon.ceph01-osd04, mon.ceph01-osd05
I have checked the time on all nodes and it is ok:
for i in ceph01-osd01 ceph01-osd02 ceph01-osd03
On Wed, Jun 10, 2015 at 4:11 PM, Pavel V. Kaygorodov wrote:
> Hi!
>
> Immediately after a reboot of mon.3 host its clock was unsynchronized and
> "clock skew detected on mon.3" warning is appeared.
> But now (more then 1 hour of uptime) the clock is synced, but the warning
> still showing.
> Is
Hi!
Immediately after a reboot of mon.3 host its clock was unsynchronized and
"clock skew detected on mon.3" warning is appeared.
But now (more then 1 hour of uptime) the clock is synced, but the warning still
showing.
Is this ok?
Or I have to restart monitor after clock synchronization?
Pavel.
On 03/13/2014 12:30 PM, Stijn De Weirdt wrote:
can we retest the clock skew condition? or get the value that the skew is?
'ceph health detail --format=json-pretty' (for instance, but 'json' or
'xml' is also allowed) will give you information on a per-monitor basis
of both skew and latency as
can we retest the clock skew condition? or get the value that the skew is?
ceph status gives
health HEALTH_WARN clock skew detected on mon.ceph003
in a polysh session (ie parallel ssh sort of thing)
ready (3)> date +%s.%N
ceph002 : 1394713567.184218678
ceph003 : 1394713567.182722045
ceph001 : 1
2014-03-13 12:59 GMT+01:00 Joao Eduardo Luis :
> Anyway, most timeouts will hold for 5 seconds. Allowing clock drifts up to
> 1 second may work, but we don't have hard data to support such claim. Over
> a second of drift may be problematic if the monitors are under some workload
> and message han
On 03/12/2014 05:04 PM, John Nielsen wrote:
On Mar 12, 2014, at 10:44 AM, Gandalf Corvotempesta
wrote:
2014-01-30 18:41 GMT+01:00 Eric Eastman :
I have this problem on some of my Ceph clusters, and I think it is due to
the older hardware the I am using does not have the best clocks. To fix
On Mar 12, 2014, at 10:44 AM, Gandalf Corvotempesta
wrote:
> 2014-01-30 18:41 GMT+01:00 Eric Eastman :
>> I have this problem on some of my Ceph clusters, and I think it is due to
>> the older hardware the I am using does not have the best clocks. To fix the
>> problem, I setup one server in my
2014-01-30 18:41 GMT+01:00 Eric Eastman :
> I have this problem on some of my Ceph clusters, and I think it is due to
> the older hardware the I am using does not have the best clocks. To fix the
> problem, I setup one server in my lab to be my local NTP time server, and
> then on each of my Ceph
I have this problem on some of my Ceph clusters, and I think it is due
to the older hardware the I am using does not have the best clocks. To
fix the problem, I setup one server in my lab to be my local NTP time
server, and then on each of my Ceph monitors, in the /etc/ntp.conf
file, I put in
2014-01-30 Emmanuel Lacour :
> here, I just wait until the skew is finished, without touching ceph. It
> doesn't seems to do anything bad ...
I've waited more than 1 hour with no success.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists
you can run 'ntpdate -b '
read ntpdate-manual for the parameters.
Markus
Am 30.01.2014 16:05, schrieb Emmanuel Lacour:
On Thu, Jan 30, 2014 at 12:53:22PM +0100, Gandalf Corvotempesta wrote:
Hi.
I'm using ntpd on each ceph server and is syncing properly but every
time that I reboot, ceph starts
On Thu, Jan 30, 2014 at 12:53:22PM +0100, Gandalf Corvotempesta wrote:
> Hi.
> I'm using ntpd on each ceph server and is syncing properly but every
> time that I reboot, ceph starts in degraded mode with "clock skew"
> warning.
>
> The only way that I have to solve this is manually restart ceph on
Hi.
I'm using ntpd on each ceph server and is syncing properly but every
time that I reboot, ceph starts in degraded mode with "clock skew"
warning.
The only way that I have to solve this is manually restart ceph on
each node (without resyncing clock)
Any suggestion ?
41 matches
Mail list logo
