Re: NTP Synchronization Setup Changes

[Mukil Kesavan]

Thanks for all the useful comments.

On Thu, Mar 31, 2016 at 11:22 PM, Brice Dutheil 
wrote:

> Hi another tip, make sure the OS doesn't come with pre-configured NTP
> synchronisation services. We had a proper NTP setup, but we missed a
> service that came with CentOS that synced to a low stratum NTP server.
>
> -- Brice
>
>
>
>
> On Thu, Mar 31, 2016 at 10:00 AM -0700, "Eric Evans"  > wrote:
>
> On Wed, Mar 30, 2016 at 8:07 PM, Mukil Kesavan wrote:
>> > Are there any issues if this causes a huge time correction on the cassandra
>> > cluster? I know that NTP gradually corrects the time on all the servers. I
>> > just wanted to understand if there were any corner cases that will cause us
>> > to lose data/schema updates when this happens. In particular, we seem to be
>> > having some issues around missing secondary indices at the moment (not all
>> > but some).
>>
>> As a thought experiment, imagine every scenario where it matters to
>> have one write occur after another (an update followed by a delete is
>> a good example).  Now imagine having your clock yanked backward to
>> correct for drift between the first such operation and the second.
>>
>> I would strongly recommend you come up with a stable NTP setup.
>>
>>
>> --
>> Eric evanseev...@wikimedia.org
>>
>>

Re: NTP Synchronization Setup Changes

[Brice Dutheil]

Hi another tip, make sure the OS doesn't come with pre-configured NTP 
synchronisation services. We had a proper NTP setup, but we missed a service 
that came with CentOS that synced to a low stratum NTP server.
-- Brice




On Thu, Mar 31, 2016 at 10:00 AM -0700, "Eric Evans"  
wrote:










On Wed, Mar 30, 2016 at 8:07 PM, Mukil Kesavan
 wrote:
> Are there any issues if this causes a huge time correction on the cassandra
> cluster? I know that NTP gradually corrects the time on all the servers. I
> just wanted to understand if there were any corner cases that will cause us
> to lose data/schema updates when this happens. In particular, we seem to be
> having some issues around missing secondary indices at the moment (not all
> but some).

As a thought experiment, imagine every scenario where it matters to
have one write occur after another (an update followed by a delete is
a good example).  Now imagine having your clock yanked backward to
correct for drift between the first such operation and the second.

I would strongly recommend you come up with a stable NTP setup.


-- 
Eric Evans
eev...@wikimedia.org

Re: NTP Synchronization Setup Changes

[Eric Evans]

On Wed, Mar 30, 2016 at 8:07 PM, Mukil Kesavan
 wrote:
> Are there any issues if this causes a huge time correction on the cassandra
> cluster? I know that NTP gradually corrects the time on all the servers. I
> just wanted to understand if there were any corner cases that will cause us
> to lose data/schema updates when this happens. In particular, we seem to be
> having some issues around missing secondary indices at the moment (not all
> but some).

As a thought experiment, imagine every scenario where it matters to
have one write occur after another (an update followed by a delete is
a good example).  Now imagine having your clock yanked backward to
correct for drift between the first such operation and the second.

I would strongly recommend you come up with a stable NTP setup.


-- 
Eric Evans
eev...@wikimedia.org

Re: NTP Synchronization Setup Changes

[Jan Kesten]

Hi Mickey,

I would strongly suggest to setup a NTP server on your site - this is not 
really a big deal and with some tutorials on the net done quickly. Then 
configure your cassandra nodes (and all the rest if you like) to use your ntp 
instead of public ones. As I have learned the hard way - cassandra is not 
really happy when nodes have different times in some cases.

Benefit of this is, that your nodes will keep time in sync even without 
connection to the internet. Of course "your time" may drift without a proper 
timesource or connection but all nodes will have the same drift and so no 
problems with consistency. If your ntp syncs your nodes will be adjusted 
smoothly.

Pro(?)-solution (what I did before): Attach a gps mouse to your ntp server and 
use that as time source. So you can have synchronized _and_ accurate time 
without any connection to public ntp servers as the gps satellites are flying 
atom clocks :)

Just my 2 cents,
Jan

Von meinem iPhone gesendet

> Am 31.03.2016 um 03:07 schrieb Mukil Kesavan :
> 
> Hi,
> 
> We run a 3 server cassandra cluster that is initially NTP synced to a single 
> physical server over LAN. This server does not have connectivity to the 
> internet for a few hours to sometimes even days. In this state we perform 
> some schema operations and reads/writes with QUORUM consistency.
> 
> Later on, the physical server has connectivity to the internet and we 
> synchronize its time to an external NTP server on the internet. 
> 
> Are there any issues if this causes a huge time correction on the cassandra 
> cluster? I know that NTP gradually corrects the time on all the servers. I 
> just wanted to understand if there were any corner cases that will cause us 
> to lose data/schema updates when this happens. In particular, we seem to be 
> having some issues around missing secondary indices at the moment (not all 
> but some).
> 
> Also, for our situation where we have to work with cassandra for a while 
> without internet connectivity, what is the preferred NTP architecture/steps 
> that have worked for you in the field?
> 
> Thanks,
> Micky

NTP Synchronization Setup Changes

[Mukil Kesavan]

Hi,

We run a 3 server cassandra cluster that is initially NTP synced to a
single physical server over LAN. This server does not have connectivity to
the internet for a few hours to sometimes even days. In this state we
perform some schema operations and reads/writes with QUORUM consistency.

Later on, the physical server has connectivity to the internet and we
synchronize its time to an external NTP server on the internet.

Are there any issues if this causes a huge time correction on the cassandra
cluster? I know that NTP gradually corrects the time on all the servers. I
just wanted to understand if there were any corner cases that will cause us
to lose data/schema updates when this happens. In particular, we seem to be
having some issues around missing secondary indices at the moment (not all
but some).

Also, for our situation where we have to work with cassandra for a while
without internet connectivity, what is the preferred NTP architecture/steps
that have worked for you in the field?

Thanks,
Micky