Re: [DISCUSS] avoid bad commits with mis-behaving clock

[Stefan Egli]

On 14/01/16 18:34, "Julian Reschke"  wrote:

>On 2016-01-14 17:36, Vikas Saurabh wrote:
>>@Julian, if I understand correctly, OAK-2682 currently is about
>> warning, right? It mentions a self-desctruct option but I think it
>> wasn't implemented.
>
>It is implemented in trunk, see r1695671 (might be only on startup,
>though).

The current model is that at startup this is enforced, but at runtime this
is not enforced at the oak level. What we currently have is a JMX method
which should be hooked into some sort of runtime monitoring (be that
external or internal eg via health checks). Such a runtime monitoring
would not be enough though as it would certainly not react fast enough.

So if we're saying we need clocks to be in sync at any given time, we
probably have to combine checking clocks upon every lease update, as well
as restricting valid revisions to be within the lease window.

Cheers,
Stefan

Re: [DISCUSS] avoid bad commits with mis-behaving clock

[Julian Reschke]

On 2016-01-14 17:36, Vikas Saurabh wrote:

Opened OAK-3883 for this. Have added following:
* We can start self-destrucut mode while updating lease
* Revision creation should check that newly created revision isn't
beyond leaseEnd time
* Implementation done for OAK-2682 might be useful
to summarize suggestions from this thread.

@Julian, if I understand correctly, OAK-2682 currently is about
warning, right? It mentions a self-desctruct option but I think it
wasn't implemented.

Thanks,
Vikas


It is implemented in trunk, see r1695671 (might be only on startup, though).

Best regards, Julian

Re: [DISCUSS] avoid bad commits with mis-behaving clock

[Vikas Saurabh]

Opened OAK-3883 for this. Have added following:
* We can start self-destrucut mode while updating lease
* Revision creation should check that newly created revision isn't
beyond leaseEnd time
* Implementation done for OAK-2682 might be useful
to summarize suggestions from this thread.

@Julian, if I understand correctly, OAK-2682 currently is about
warning, right? It mentions a self-desctruct option but I think it
wasn't implemented.

Thanks,
Vikas

Re: [DISCUSS] avoid bad commits with mis-behaving clock

[Marcel Reutegger]

Hi,

On 14/01/16 01:26, "Vikas Saurabh" wrote:
>There's a bit of respite on trunk (due to [1] and friends) and a
>sequence of events like {{t1 -> update lease}}, {{t2=t1+7.5hrs ->
>commit}} would lead to a shutdown as a big jump like 7.5 hours would
>be beyond lease end time. BUT, if there's lease update before a commit
>after a clock jump, then even on trunk there's no safeguard.
>If we update lease update logic to shut down as well in case it can't
>update lease for a long time then an instance with forward jumping
>clock would self-destruct.

Please note, this is not sufficient. We should also introduce a
a check when a commit revision is created. The revision timestamp
must always be less than the leaseEnd time. In addition with your
above proposal, that should prevent commits with a future revision
caused by clock jumps.

Could you please create an issue for this?

>BTW, the approach mentioned above saves only agains a clock which
>jumps ahead directly. We'd still have similar issue for a clock slowly
>skews forward. But, even then I think it's worth it to have to solve
>the particular case.

this is probably less of an issue, because you'd notice it at some
point. Commits on the cluster nodes with the correct clock would
get delayed and WARN messages appear in the logs.

Regards
 Marcel

Re: [DISCUSS] avoid bad commits with mis-behaving clock

[Julian Reschke]

On 2016-01-14 01:26, Vikas Saurabh wrote:

...


Note that in trunk, we also have mechanism to detect misbehaving system 
clocks; see .


Best regards, Julian