Re: [Openstack] [Mirantis] How to keep ntpd down

[John Fleming]

Don't know if you've tried this as a bandaid changing the clock source might 
help. I recently found a small group of servers that chose tsc as the clock 
source. While using tsc the system clock would lose 1 second every 30 seconds. 
Switching to hpet as the clock source (well technically all the others worked) 
would stabilize the clock and thus ntpd wouldn't lose its mind. 
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Mirantis] How to keep ntpd down

[Jeremy Stanley]

On 2017-07-21 05:56:48 +0530 (+0530), Raja T Nair wrote:
[...]
> Many other servers sync with this one too. Also only one
> controller had issues with time. Kind of stuck here, as I have no
> idea why one node's ntpd would fail :(
[...]

I've seen this from time to time over the years and it's nearly
always a bad RTC. Poor quality control is not unusual since the
manufacturers assume you either won't care or will use something
like NTP to keep things in reasonable sync, but sometimes you'll get
one which is well outside your ntpd's tolerance for drift and it'll
just flat refuse to update it. In those sorts of situations I've
pretty much always sent the server back or requested appropriate
replacement parts to service it myself on site.
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Mirantis] How to keep ntpd down

[John Petrini]

To Brad's point - if your controllers are VM's you might also want to have
a look at Chrony https://chrony.tuxfamily.org/. It's supposed to perform
much better on virtual machines.

___

John Petrini

On Thu, Jul 20, 2017 at 9:20 PM, Brad Knowles 
wrote:

> On Jul 20, 2017, at 7:26 PM, Raja T Nair  wrote:
>
> > Thanks a lot for the reply, John.
> >
> > Yes I understand that time is really important for cluster setup, that's
> why I was panicking and looking for alternatives when I found time drifting
> while ntpd was still on.
> > So I was planning to do a ``ntpdate w.x.y.z '' every 2 mins in order to
> keep time in sync.
> >
> > Would want to investigate this. My upstream time server seems fine, its
> on a baremetal. Many other servers sync with this one too. Also only one
> controller had issues with time.
> > Kind of stuck here, as I have no idea why one node's ntpd would fail :(
>
> Doing a cron job with ntpdate will cause your time to bounce all over the
> place, and that will be even worse than what you've had so far.
>
> I've been a member of the NTP Public Services Project since 2003, and I've
> seen a lot of NTP problems over the years, especially on virtual machines.
> Historically, our advice was to not even run ntpd at all on a VM, but
> instead to run it on the bare hardware underneath, and then make sure that
> you're running the necessary hooks in the hypervisor and the guest OSes to
> pass good quality time up the stack to all the clients.
>
> I'm not sure if that is still the best advice or not -- I think it may
> depend on your hypervisor and your guest OSes.
>
> But if you do run ntpd on the guests, there are things you can do to
> measure larger-than-normal amounts of drift and compensate for that.  I
> would direct you to the mailing list questi...@lists.ntp.org for more
> information.
>
> --
> Brad Knowles 
>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Mirantis] How to keep ntpd down

[Brad Knowles]

On Jul 20, 2017, at 7:26 PM, Raja T Nair  wrote:

> Thanks a lot for the reply, John.
> 
> Yes I understand that time is really important for cluster setup, that's why 
> I was panicking and looking for alternatives when I found time drifting while 
> ntpd was still on.
> So I was planning to do a ``ntpdate w.x.y.z '' every 2 mins in order to keep 
> time in sync.
> 
> Would want to investigate this. My upstream time server seems fine, its on a 
> baremetal. Many other servers sync with this one too. Also only one 
> controller had issues with time.
> Kind of stuck here, as I have no idea why one node's ntpd would fail :(

Doing a cron job with ntpdate will cause your time to bounce all over the 
place, and that will be even worse than what you've had so far.

I've been a member of the NTP Public Services Project since 2003, and I've seen 
a lot of NTP problems over the years, especially on virtual machines.  
Historically, our advice was to not even run ntpd at all on a VM, but instead 
to run it on the bare hardware underneath, and then make sure that you're 
running the necessary hooks in the hypervisor and the guest OSes to pass good 
quality time up the stack to all the clients.

I'm not sure if that is still the best advice or not -- I think it may depend 
on your hypervisor and your guest OSes.

But if you do run ntpd on the guests, there are things you can do to measure 
larger-than-normal amounts of drift and compensate for that.  I would direct 
you to the mailing list questi...@lists.ntp.org for more information.

--
Brad Knowles 



signature.asc
Description: Message signed with OpenPGP
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Mirantis] How to keep ntpd down

[Raja T Nair]

Thanks a lot for the reply, John.

Yes I understand that time is really important for cluster setup, that's
why I was panicking and looking for alternatives when I found time drifting
while ntpd was still on.
So I was planning to do a ``ntpdate w.x.y.z '' every 2 mins in order to
keep time in sync.

Would want to investigate this. My upstream time server seems fine, its on
a baremetal. Many other servers sync with this one too. Also only one
controller had issues with time.
Kind of stuck here, as I have no idea why one node's ntpd would fail :(

Regards,
Raja.



On 20 July 2017 at 16:27, John Petrini  wrote:

> On all of the controllers? crm resource stop clone_p_ntp should do it.
> Although I can't imagine why you would want to do this. Time is very
> important in OpenStack (and Ceph if you are running it) which it sounds
> like you've already found out.
>
> The whole purpose of NTP is to keep your time in sync - if it's not doing
> that you should be looking for the root cause not disabling it. You might
> want to start by looking at your upstream time servers that the controllers
> are using. This is configured in Fuel and the configuration is stored in
> /etc/npt.conf on the controllers.
>
> I'd highly recommend setting up monitoring of ntp so you know when the
> clock starts to drift and can respond to it before it drifts too far and
> becomes a problem.
>
> ___
>
> John Petrini
>
>
> On Thu, Jul 20, 2017 at 6:29 AM, Raja T Nair  wrote:
>
>> Hello All,
>>
>> Mirantis 7.0
>>
>> I am trying to keep ntpd down and do a periodic ntpdate against a time
>> server.
>> This is because one of the controllers started to drift and services in
>> that not started to go down.
>>
>> But it seems that the ntpd daemon comes up after 10 sec every time i stop
>> it.
>> Is there a monitor running somewhere which does brings it back?
>>
>> Please guide me on this and also tell me if I am doing something wrong.
>>
>> Regards,
>> Raja.
>>
>> --
>> :^)
>>
>> ___
>> Mailing list: http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>> Post to : openstack@lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>>
>>
>


-- 
:^)
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Mirantis] How to keep ntpd down

[John Petrini]

On all of the controllers? crm resource stop clone_p_ntp should do it.
Although I can't imagine why you would want to do this. Time is very
important in OpenStack (and Ceph if you are running it) which it sounds
like you've already found out.

The whole purpose of NTP is to keep your time in sync - if it's not doing
that you should be looking for the root cause not disabling it. You might
want to start by looking at your upstream time servers that the controllers
are using. This is configured in Fuel and the configuration is stored in
/etc/npt.conf on the controllers.

I'd highly recommend setting up monitoring of ntp so you know when the
clock starts to drift and can respond to it before it drifts too far and
becomes a problem.

___

John Petrini


On Thu, Jul 20, 2017 at 6:29 AM, Raja T Nair  wrote:

> Hello All,
>
> Mirantis 7.0
>
> I am trying to keep ntpd down and do a periodic ntpdate against a time
> server.
> This is because one of the controllers started to drift and services in
> that not started to go down.
>
> But it seems that the ntpd daemon comes up after 10 sec every time i stop
> it.
> Is there a monitor running somewhere which does brings it back?
>
> Please guide me on this and also tell me if I am doing something wrong.
>
> Regards,
> Raja.
>
> --
> :^)
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

[Openstack] [Mirantis] How to keep ntpd down

[Raja T Nair]

Hello All,

Mirantis 7.0

I am trying to keep ntpd down and do a periodic ntpdate against a time
server.
This is because one of the controllers started to drift and services in
that not started to go down.

But it seems that the ntpd daemon comes up after 10 sec every time i stop
it.
Is there a monitor running somewhere which does brings it back?

Please guide me on this and also tell me if I am doing something wrong.

Regards,
Raja.

-- 
:^)
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack