Re: How does cassandra achieve Linearizability?
;>> > >>>>>> -- > >>>>>> Kind regards, > >>>>>> Michael > >>>>>> > >>>>>> On 02/09/2017 06:47 PM, > >>>>>> Kant Kodali wrote: > >>>>>> > Hi Justin, > >>>>>> > > >>>>>> > There are bunch of > >>>>>> issues w.r.t to > >>>>>> synchronization of clocks > >>>>>> when we > >>>>>> > used ntpd. Also the time > >>>>>> it took to sync the clocks > >>>>>> was approx 140ms > >>>>>> > (don't quote me on it > >>>>>> though because it is > >>>>>> reported by our devops :) > >>>>>> > > >>>>>> > we have multiple clients > >>>>>> (for example bunch of > >>>>>> micro services are > >>>>>> > reading from Cassandra) > >>>>>> I am not sure how one can > >>>>>> achieve > >>>>>> > Linearizability by > >>>>>> setting timestamps on the > >>>>>> clients ? since there is no > >>>>>> > total ordering across > >>>>>> multiple clients. > >>>>>> > > >>>>>> > Thanks! > >>>>>> > > >>>>>> > > >>>>>> > On Thu, Feb 9, 2017 at > >>>>>> 4:16 PM, Justin Cameron > >>>>>> >>>>>> jus...@instaclustr.com> > >>>>>> > jus...@instaclustr.com > >>>>>> jus...@instaclustr.com>>> > >>>>>> wrote: > >>>>>> > > >>>>>> > Hi Kant, > >>>>>> > > >>>>>> > Clock > >>>>>> synchronization is > >>>>>> important - you should > >>>>>> ensure that ntpd is > >>>>>> > properly configured > >>>>>> on all nodes. If your > >>>>>> particular use case is > >>>>>> > especially sensitive > >>>>>> to out-of-order mutations > >>>>>> it is possible to set > >>>>>> > timestamps on the > >>>>>> client side using the > >>>>>>
Re: How does cassandra achieve Linearizability?
setting up GPS modules and how >>>>>> that can be helpful. Let me >>>>>> research on that but good point. >>>>>> >>>>>> On Thu, Feb 9, 2017 at 5:09 >>>>>> PM, Michael Shuler >>>>>> >>>>> <mailto:mich...@pbandjelly.org>> >>>>>> wrote: >>>>>> >>>>>> If you require the best >>>>>> precision you can get, >>>>>> setting up a pair of >>>>>> stratum 1 ntpd masters in >>>>>> each data center location >>>>>> with a GPS modules >>>>>> is not terribly complex. >>>>>> Low latency and jitter on >>>>>> servers you manage. >>>>>> 140ms is a long way away >>>>>> network-wise, and I would >>>>>> suggest that was a >>>>>> poor choice of upstream >>>>>> (probably stratum 2 or 3) >>>>>> source. >>>>>> >>>>>> As Jonathan mentioned, >>>>>> there's no guarantee from >>>>>> Cassandra, but if you >>>>>> need as close as you can >>>>>> get, you'll probably need >>>>>> to do it yourself. >>>>>> >>>>>> (I run several stratum 2 >>>>>> ntpd servers for >>>>>> pool.ntp.org >>>>>> <http://pool.ntp.org/>) >>>>>> >>>>>> -- >>>>>> Kind regards, >>>>>> Michael >>>>>> >>>>>> On 02/09/2017 06:47 PM, >>>>>> Kant Kodali wrote: >>>>>> > Hi Justin, >>>>>> > >>>>>> > There are bunch of >>>>>> issues w.r.t to >>>>>> synchronization of clocks >>>>>> when we >>>>>> > used ntpd. Also the time >>>>>> it took to sync the clocks >>>>>> was approx 140ms >>>>>> > (don't quote me on it >>>>>> though because it is >>>>>> reported by our devops :) >>>>>> > >>>>>> > we have multiple clients >>>>>> (for example bunch of >>>>>> micro services are >>>>>> > reading from Cassandra) >>>>>> I am not sure how one can >>>>>>
Re: How does cassandra achieve Linearizability?
of >> stratum 1 ntpd masters in each data center location with a GPS modules >> is not terribly complex. Low latency and jitter on servers you manage. >> 140ms is a long way away network-wise, and I would suggest that was a >> poor choice of upstream (probably stratum 2 or 3) source. >> >> As Jonathan mentioned, there's no guarantee from Cassandra, but if you >> need as close as you can get, you'll probably need to do it yourself. >> >> (I run several stratum 2 ntpd servers for pool.ntp.org) >> >> -- >> Kind regards, >> Michael >> >> On 02/09/2017 06:47 PM, Kant Kodali wrote: >> > Hi Justin, >> > >> > There are bunch of issues w.r.t to synchronization of clocks when we >> > used ntpd. Also the time it took to sync the clocks was approx 140ms >> > (don't quote me on it though because it is reported by our devops :) >> > >> > we have multiple clients (for example bunch of micro services are >> > reading from Cassandra) I am not sure how one can achieve >> > Linearizability by setting timestamps on the clients ? since there is no >> > total ordering across multiple clients. >> > >> > Thanks! >> > >> > >> > On Thu, Feb 9, 2017 at 4:16 PM, Justin Cameron > > <mailto:jus...@instaclustr.com>> wrote: >> > >> > Hi Kant, >> > >> > Clock synchronization is important - you should ensure that ntpd is >> > properly configured on all nodes. If your particular use case is >> > especially sensitive to out-of-order mutations it is possible to set >> > timestamps on the client side using the >> > drivers. https://docs.datastax.com/en/d >> eveloper/java-driver/3.1/manual/query_timestamps/ >> > <https://docs.datastax.com/en/developer/java-driver/3.1/man >> ual/query_timestamps/> >> > >> > We use our own NTP cluster to reduce clock drift as much as >> > possible, but public NTP servers are good enough for most >> > uses. https://www.instaclustr.com/blog/2015/11/05/apache-cassandra >> -synchronization/ >> > <https://www.instaclustr.com/blog/2015/11/05/apache-cassand >> ra-synchronization/> >> > >> > Cheers, >> > Justin >> > >> > On Thu, 9 Feb 2017 at 16:09 Kant Kodali > > <mailto:k...@peernova.com>> wrote: >> > >> > How does Cassandra achieve Linearizability with “Last write >> > wins” (conflict resolution methods based on time-of-day clocks) >> ? >> > >> > Relying on synchronized clocks are almost certainly >> > non-linearizable, because clock timestamps cannot be guaranteed >> > to be consistent with actual event ordering due to clock skew. >> > isn't it? >> > >> > Thanks! >> > >> > -- >> > >> > Justin Cameron >> > >> > Senior Software Engineer | Instaclustr >> > >> > >> > >> > >> > This email has been sent on behalf of Instaclustr Pty Ltd >> > (Australia) and Instaclustr Inc (USA). >> > >> > This email and any attachments may contain confidential and legally >> > privileged information. If you are not the intended recipient, do >> > not copy or disclose its content, but please reply to this email >> > immediately and highlight the error to the sender and then >> > immediately delete the message. >> > >> > >> >> >> >> >> >> >> >> >> >> -- >> Benjamin Roth >> Prokurist >> >> Jaumo GmbH · www.jaumo.com >> Wehrstraße 46 · 73035 Göppingen · Germany >> Phone +49 7161 304880-6 <+49%207161%203048806> · Fax +49 7161 304880-1 >> <+49%207161%203048801> >> AG Ulm · HRB 731058 · Managing Director: Jens Kammerer >> >> >> >> >> >> >> One thing that always bothered me: Intelligent clients and dynamic snitch >> are designed to attempt to route requests to the same node to attempt to >> take advantage of cache pinning etc. You would think under these conditions >> one could naturally elect a "leader" for a "group" of keys that could >> persist for a few hundred milliseconds and batch up the round trips for a >> number of operations. Maybe that is what the distinguished coordinator is >> in some regards. >> >> >> >> > My two cents: The current issue is "feature complete" and the author > stated ready for review 2 years ago. But I can see that as the issue stands > it forces some hard choices to be made concerning the migration path and in > depth code changes. > > Also I think there is some question (in my mind) as to how we ensure some > of the subtle contracted/non contracted semantics stay in place. As in they > work a "certain way" and how confident is everyone that a "better way" does > not end up causing some pain for someone using it currently. I assume this > as a common case where a feature request is not being engaged with. >
Re: How does cassandra achieve Linearizability?
e is no > > total ordering across multiple clients. > > > > Thanks! > > > > > > On Thu, Feb 9, 2017 at 4:16 PM, Justin Cameron > <mailto:jus...@instaclustr.com>> wrote: > > > > Hi Kant, > > > > Clock synchronization is important - you should ensure that ntpd is > > properly configured on all nodes. If your particular use case is > > especially sensitive to out-of-order mutations it is possible to set > > timestamps on the client side using the > > drivers. https://docs.datastax.com/en/d > eveloper/java-driver/3.1/manual/query_timestamps/ > > <https://docs.datastax.com/en/developer/java-driver/3.1/man > ual/query_timestamps/> > > > > We use our own NTP cluster to reduce clock drift as much as > > possible, but public NTP servers are good enough for most > > uses. https://www.instaclustr.com/blog/2015/11/05/apache-cassandra > -synchronization/ > > <https://www.instaclustr.com/blog/2015/11/05/apache-cassand > ra-synchronization/> > > > > Cheers, > > Justin > > > > On Thu, 9 Feb 2017 at 16:09 Kant Kodali > <mailto:k...@peernova.com>> wrote: > > > > How does Cassandra achieve Linearizability with “Last write > > wins” (conflict resolution methods based on time-of-day clocks) ? > > > > Relying on synchronized clocks are almost certainly > > non-linearizable, because clock timestamps cannot be guaranteed > > to be consistent with actual event ordering due to clock skew. > > isn't it? > > > > Thanks! > > > > -- > > > > Justin Cameron > > > > Senior Software Engineer | Instaclustr > > > > > > > > > > This email has been sent on behalf of Instaclustr Pty Ltd > > (Australia) and Instaclustr Inc (USA). > > > > This email and any attachments may contain confidential and legally > > privileged information. If you are not the intended recipient, do > > not copy or disclose its content, but please reply to this email > > immediately and highlight the error to the sender and then > > immediately delete the message. > > > > > > > > > > > > > > -- > Benjamin Roth > Prokurist > > Jaumo GmbH · www.jaumo.com > Wehrstraße 46 · 73035 Göppingen · Germany > Phone +49 7161 304880-6 <+49%207161%203048806> · Fax +49 7161 304880-1 > <+49%207161%203048801> > AG Ulm · HRB 731058 · Managing Director: Jens Kammerer > > > > > > > One thing that always bothered me: Intelligent clients and dynamic snitch > are designed to attempt to route requests to the same node to attempt to > take advantage of cache pinning etc. You would think under these conditions > one could naturally elect a "leader" for a "group" of keys that could > persist for a few hundred milliseconds and batch up the round trips for a > number of operations. Maybe that is what the distinguished coordinator is > in some regards. > > > > My two cents: The current issue is "feature complete" and the author stated ready for review 2 years ago. But I can see that as the issue stands it forces some hard choices to be made concerning the migration path and in depth code changes. Also I think there is some question (in my mind) as to how we ensure some of the subtle contracted/non contracted semantics stay in place. As in they work a "certain way" and how confident is everyone that a "better way" does not end up causing some pain for someone using it currently. I assume this as a common case where a feature request is not being engaged with.
Re: How does cassandra achieve Linearizability?
;>> * accepted. So in practice, which timestamp we use for a >>>>>>>>>>>given proposal doesn't affect correctness but it does >>>>>>>>>>> * affect the chance of making progress (if we pick a >>>>>>>>>>>timestamp lower than what has been proposed before, our >>>>>>>>>>> * new proposal will just get rejected). >>>>>>>>>>> >>>>>>>>>>> Effectively paxos removes the ability to use custom >>>>>>>>>>> timestamps and addresses clock variance by rejecting ballots >>>>>>>>>>> with timestamps less than what was last seen. You can learn >>>>>>>>>>> more by reading through the other comments and code in that >>>>>>>>>>> file. >>>>>>>>>>> >>>>>>>>>>> Last write wins is a free for all that guarantees you >>>>>>>>>>> *nothing* except the timestamp is used as a tiebreaker. >>>>>>>>>>> Here we acknowledge things like the speed of light as being >>>>>>>>>>> a real problem that isn’t going away anytime soon. This >>>>>>>>>>> problem is sometimes addressed with event sourcing rather >>>>>>>>>>> than mutating in place. >>>>>>>>>>> >>>>>>>>>>> Hope this helps. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Jon >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On Feb 9, 2017, at 5:21 PM, Kant Kodali >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> @Justin I read this article >>>>>>>>>>>> http://www.datastax.com/dev/blog/lightweight-transactions-in-cassandra-2-0. >>>>>>>>>>>> And it clearly says Linearizable consistency can be >>>>>>>>>>>> achieved with LWT's. so should I assume the >>>>>>>>>>>> Linearizability in the context of the above article is >>>>>>>>>>>> possible with LWT's and synchronization of clocks through >>>>>>>>>>>> ntpd ? because LWT's also follow Last Write Wins. isn't it? >>>>>>>>>>>> Also another question does most of the production clusters >>>>>>>>>>>> do setup ntpd? If so what is the time it takes to sync? any >>>>>>>>>>>> idea >>>>>>>>>>>> >>>>>>>>>>>> @Micheal Schuler Are you referring to something like true >>>>>>>>>>>> time as in >>>>>>>>>>>> https://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf? >>>>>>>>>>>> Actually I never heard of setting up GPS modules and how >>>>>>>>>>>> that can be helpful. Let me research on that but good >>>>>>>>>>>> point. >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Feb 9, 2017 at 5:09 PM, Michael Shuler >>>>>>>>>>>> wrote: >>>>>>>>>>>>> If you require the best precision you can get, setting up >>>>>>>>>>>>> a pair of >>>>>>>>>>>>> stratum 1 ntpd masters in each data center location with a >>>>>>>>>>>>> GPS modules >>>>>>>>>>>>> is not terribly complex. Low latency and jitter on servers >>>>>>>>>>>>> you manage. >>>>>>>>>>>>> 140ms is a long way away network-wise, and I would suggest >>>>>>>>>>>>> that was a >>>>>>>>>>>>> poor choice of upstream (probably stratum 2 or 3) source. >>>>>>>>>>>>> >>>>>>>>>>>&g
Re: How does cassandra achieve Linearizability?
ve. Are there any plans to make this better? > > On Fri, Feb 10, 2017 at 12:17 AM, Kant Kodali wrote: > > Hi Jon, > > Thanks a lot for your response. I am well aware that the LWW != LWT but I > was talking more in terms of LWW with respective to LWT's which I believe > you answered. so thanks much! > > > kant > > > On Thu, Feb 9, 2017 at 6:01 PM, Jon Haddad > wrote: > > LWT != Last Write Wins. They are totally different. > > LWTs give you (assuming you also read at SERIAL) “atomic consistency”, > meaning you are able to perform operations atomically and in isolation. > That’s the safety blanket everyone wants but is extremely expensive, > especially in Cassandra. The lightweight part, btw, may be a little > optimistic, especially if a key is under contention. With regard to the > “last write” part you’re asking about - w/ LWT Cassandra provides the > timestamp and manages it as part of the ballot, and it always is > increasing. See > org.apache.cassandra.service.ClientState#getTimestampForPaxos. > From the code: > > * Returns a timestamp suitable for paxos given the timestamp of the last > known commit (or in progress update). > * Paxos ensures that the timestamp it uses for commits respects the > serial order of those commits. It does so > * by having each replica reject any proposal whose timestamp is not > strictly greater than the last proposal it > * accepted. So in practice, which timestamp we use for a given proposal > doesn't affect correctness but it does > * affect the chance of making progress (if we pick a timestamp lower than > what has been proposed before, our > * new proposal will just get rejected). > > Effectively paxos removes the ability to use custom timestamps and > addresses clock variance by rejecting ballots with timestamps less than > what was last seen. You can learn more by reading through the other > comments and code in that file. > > Last write wins is a free for all that guarantees you *nothing* except the > timestamp is used as a tiebreaker. Here we acknowledge things like the > speed of light as being a real problem that isn’t going away anytime soon. > This problem is sometimes addressed with event sourcing rather than > mutating in place. > > Hope this helps. > > > Jon > > > > > On Feb 9, 2017, at 5:21 PM, Kant Kodali wrote: > > @Justin I read this article http://www.datastax.com/dev/bl > og/lightweight-transactions-in-cassandra-2-0. And it clearly says > Linearizable consistency can be achieved with LWT's. so should I assume > the Linearizability in the context of the above article is possible with > LWT's and synchronization of clocks through ntpd ? because LWT's also > follow Last Write Wins. isn't it? Also another question does most of the > production clusters do setup ntpd? If so what is the time it takes to sync? > any idea > > @Micheal Schuler Are you referring to something like true time as in > https://static.googleusercontent.com/media/research.google.c > om/en//archive/spanner-osdi2012.pdf? Actually I never heard of setting > up GPS modules and how that can be helpful. Let me research on that but > good point. > > On Thu, Feb 9, 2017 at 5:09 PM, Michael Shuler > wrote: > > If you require the best precision you can get, setting up a pair of > stratum 1 ntpd masters in each data center location with a GPS modules > is not terribly complex. Low latency and jitter on servers you manage. > 140ms is a long way away network-wise, and I would suggest that was a > poor choice of upstream (probably stratum 2 or 3) source. > > As Jonathan mentioned, there's no guarantee from Cassandra, but if you > need as close as you can get, you'll probably need to do it yourself. > > (I run several stratum 2 ntpd servers for pool.ntp.org) > > -- > Kind regards, > Michael > > On 02/09/2017 06:47 PM, Kant Kodali wrote: > > Hi Justin, > > > > There are bunch of issues w.r.t to synchronization of clocks when we > > used ntpd. Also the time it took to sync the clocks was approx 140ms > > (don't quote me on it though because it is reported by our devops :) > > > > we have multiple clients (for example bunch of micro services are > > reading from Cassandra) I am not sure how one can achieve > > Linearizability by setting timestamps on the clients ? since there is no > > total ordering across multiple clients. > > > > Thanks! > > > > > > On Thu, Feb 9, 2017 at 4:16 PM, Justin Cameron > <mailto:jus...@instaclustr.com>> wrote: > > > > Hi Kant, > > > > Clock synchronization is important - you should ensure that ntpd i
Re: How does cassandra achieve Linearizability?
t;>>>>>> On Feb 9, 2017, at 5:21 PM, Kant Kodali >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> @Justin I read this article >>>>>>>>>> http://www.datastax.com/dev/blog/lightweight-transactions-in-cassandra-2-0. >>>>>>>>>> And it clearly says Linearizable consistency can be achieved >>>>>>>>>> with LWT's. so should I assume the Linearizability in the >>>>>>>>>> context of the above article is possible with LWT's and >>>>>>>>>> synchronization of clocks through ntpd ? because LWT's also >>>>>>>>>> follow Last Write Wins. isn't it? Also another question does >>>>>>>>>> most of the production clusters do setup ntpd? If so what is >>>>>>>>>> the time it takes to sync? any idea >>>>>>>>>> >>>>>>>>>> @Micheal Schuler Are you referring to something like true >>>>>>>>>> time as in >>>>>>>>>> https://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf? >>>>>>>>>> Actually I never heard of setting up GPS modules and how that >>>>>>>>>> can be helpful. Let me research on that but good point. >>>>>>>>>> >>>>>>>>>> On Thu, Feb 9, 2017 at 5:09 PM, Michael Shuler >>>>>>>>>> wrote: >>>>>>>>>>> If you require the best precision you can get, setting up a >>>>>>>>>>> pair of >>>>>>>>>>> stratum 1 ntpd masters in each data center location with a >>>>>>>>>>> GPS modules >>>>>>>>>>> is not terribly complex. Low latency and jitter on servers >>>>>>>>>>> you manage. >>>>>>>>>>> 140ms is a long way away network-wise, and I would suggest >>>>>>>>>>> that was a >>>>>>>>>>> poor choice of upstream (probably stratum 2 or 3) source. >>>>>>>>>>> >>>>>>>>>>> As Jonathan mentioned, there's no guarantee from Cassandra, >>>>>>>>>>> but if you >>>>>>>>>>> need as close as you can get, you'll probably need to do it >>>>>>>>>>> yourself. >>>>>>>>>>> >>>>>>>>>>> (I run several stratum 2 ntpd servers for pool.ntp.org[2]) >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Kind regards, Michael >>>>>>>>>>> >>>>>>>>>>> On 02/09/2017 06:47 PM, Kant Kodali wrote: >>>>>>>>>>> > Hi Justin, >>>>>>>>>>> > >>>>>>>>>>> > There are bunch of issues w.r.t to synchronization of >>>>>>>>>>> > clocks when we used ntpd. Also the time it took to sync >>>>>>>>>>> > the clocks was approx 140ms (don't quote me on it though >>>>>>>>>>> > because it is reported by our devops :) >>>>>>>>>>> > >>>>>>>>>>> > we have multiple clients (for example bunch of micro >>>>>>>>>>> > services are reading from Cassandra) I am not sure how >>>>>>>>>>> > one can achieve Linearizability by setting timestamps on >>>>>>>>>>> > the clients ? since there is no total ordering across >>>>>>>>>>> > multiple clients. >>>>>>>>>>> > >>>>>>>>>>> > Thanks! >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > On Thu, Feb 9, 2017 at 4:16 PM, Justin Cameron >>>>>>>>>>> > >>>>>>>>>> > <mailto:jus...@instaclustr.com>> wrote: >>>>>>>>>>> > >>>>>>>>>>> > Hi Kant, >>>>>>>>>>> > >>>>>>>>>>> > Clock synchronization is important - you should >>>>>>>>>>> > ensure that ntpd is properly configured on all nodes. >>>>>>>>>>> > If your particular use case is especially sensitive >>>>>>>>>>> > to out-of-order mutations it is possible to set >>>>>>>>>>> > timestamps on the client side using the drivers. >>>>>>>>>>> > >>>>>>>>>>> https://docs.datastax.com/en/developer/java-driver/3.1/manual/query_timestamps/ >>>>>>>>>>> > >>>>>>>>>>> <https://docs.datastax.com/en/developer/java-driver/3.1/manual/query_timestamps/> >>>>>>>>>>> > >>>>>>>>>>> > We use our own NTP cluster to reduce clock drift as >>>>>>>>>>> > much as possible, but public NTP servers are good >>>>>>>>>>> > enough for most uses. >>>>>>>>>>> > >>>>>>>>>>> https://www.instaclustr.com/blog/2015/11/05/apache-cassandra-synchronization/ >>>>>>>>>>> > >>>>>>>>>>> <https://www.instaclustr.com/blog/2015/11/05/apache-cassandra-synchronization/> >>>>>>>>>>> > >>>>>>>>>>> > Cheers, Justin >>>>>>>>>>> > >>>>>>>>>>> > On Thu, 9 Feb 2017 at 16:09 Kant Kodali >>>>>>>>>>> > >>>>>>>>>> > <mailto:k...@peernova.com>> wrote: >>>>>>>>>>> > >>>>>>>>>>> > How does Cassandra achieve Linearizability with >>>>>>>>>>> > “Last write wins” (conflict resolution methods >>>>>>>>>>> > based on time-of-day clocks) ? >>>>>>>>>>> > >>>>>>>>>>> > Relying on synchronized clocks are almost >>>>>>>>>>> > certainly non-linearizable, because clock >>>>>>>>>>> > timestamps cannot be guaranteed to be consistent >>>>>>>>>>> > with actual event ordering due to clock skew. >>>>>>>>>>> > isn't it? >>>>>>>>>>> > >>>>>>>>>>> > Thanks! >>>>>>>>>>> > >>>>>>>>>>> > -- >>>>>>>>>>> > >>>>>>>>>>> > Justin Cameron >>>>>>>>>>> > >>>>>>>>>>> > Senior Software Engineer | Instaclustr >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > This email has been sent on behalf of Instaclustr Pty >>>>>>>>>>> > Ltd >>>>>>>>>>> > (Australia) and Instaclustr Inc (USA). >>>>>>>>>>> > >>>>>>>>>>> > This email and any attachments may contain >>>>>>>>>>> > confidential and legally >>>>>>>>>>> > privileged information. If you are not the intended >>>>>>>>>>> > recipient, do >>>>>>>>>>> > not copy or disclose its content, but please reply to >>>>>>>>>>> > this email >>>>>>>>>>> > immediately and highlight the error to the sender and >>>>>>>>>>> > then >>>>>>>>>>> > immediately delete the message. >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Benjamin Roth >>>>>> Prokurist >>>>>> >>>>>> Jaumo GmbH · www.jaumo.com >>>>>> Wehrstraße 46 · 73035 Göppingen · Germany >>>>>> Phone +49 7161 304880-6[3] · Fax +49 7161 304880-1[4] >>>>>> AG Ulm · HRB 731058 · Managing Director: Jens Kammerer >>>>>> >>>>>> >>>>> >>>>> >>>>> >> > One thing that always bothered me: Intelligent clients and dynamic > snitch are designed to attempt to route requests to the same node to > attempt to take advantage of cache pinning etc. You would think under > these conditions one could naturally elect a "leader" for a "group" of > keys that could persist for a few hundred milliseconds and batch up > the round trips for a number of operations. Maybe that is what the > distinguished coordinator is in some regards. Links: 1. https://issues.apache.org/jira/browse/CASSANDRA-6246?jql=text%20~%20%22epaxos%22 2. http://pool.ntp.org/ 3. tel:+49%207161%203048806 4. tel:+49%207161%203048801
Re: How does cassandra achieve Linearizability?
for a given proposal > doesn't affect correctness but it does > * affect the chance of making progress (if we pick a timestamp lower than > what has been proposed before, our > * new proposal will just get rejected). > > Effectively paxos removes the ability to use custom timestamps and > addresses clock variance by rejecting ballots with timestamps less than > what was last seen. You can learn more by reading through the other > comments and code in that file. > > Last write wins is a free for all that guarantees you *nothing* except the > timestamp is used as a tiebreaker. Here we acknowledge things like the > speed of light as being a real problem that isn’t going away anytime soon. > This problem is sometimes addressed with event sourcing rather than > mutating in place. > > Hope this helps. > > > Jon > > > > > On Feb 9, 2017, at 5:21 PM, Kant Kodali wrote: > > @Justin I read this article http://www.datastax.com/dev/bl > og/lightweight-transactions-in-cassandra-2-0. And it clearly says > Linearizable consistency can be achieved with LWT's. so should I assume > the Linearizability in the context of the above article is possible with > LWT's and synchronization of clocks through ntpd ? because LWT's also > follow Last Write Wins. isn't it? Also another question does most of the > production clusters do setup ntpd? If so what is the time it takes to sync? > any idea > > @Micheal Schuler Are you referring to something like true time as in > https://static.googleusercontent.com/media/research.google.c > om/en//archive/spanner-osdi2012.pdf? Actually I never heard of setting > up GPS modules and how that can be helpful. Let me research on that but > good point. > > On Thu, Feb 9, 2017 at 5:09 PM, Michael Shuler > wrote: > > If you require the best precision you can get, setting up a pair of > stratum 1 ntpd masters in each data center location with a GPS modules > is not terribly complex. Low latency and jitter on servers you manage. > 140ms is a long way away network-wise, and I would suggest that was a > poor choice of upstream (probably stratum 2 or 3) source. > > As Jonathan mentioned, there's no guarantee from Cassandra, but if you > need as close as you can get, you'll probably need to do it yourself. > > (I run several stratum 2 ntpd servers for pool.ntp.org) > > -- > Kind regards, > Michael > > On 02/09/2017 06:47 PM, Kant Kodali wrote: > > Hi Justin, > > > > There are bunch of issues w.r.t to synchronization of clocks when we > > used ntpd. Also the time it took to sync the clocks was approx 140ms > > (don't quote me on it though because it is reported by our devops :) > > > > we have multiple clients (for example bunch of micro services are > > reading from Cassandra) I am not sure how one can achieve > > Linearizability by setting timestamps on the clients ? since there is no > > total ordering across multiple clients. > > > > Thanks! > > > > > > On Thu, Feb 9, 2017 at 4:16 PM, Justin Cameron > <mailto:jus...@instaclustr.com>> wrote: > > > > Hi Kant, > > > > Clock synchronization is important - you should ensure that ntpd is > > properly configured on all nodes. If your particular use case is > > especially sensitive to out-of-order mutations it is possible to set > > timestamps on the client side using the > > drivers. https://docs.datastax.com/en/d > eveloper/java-driver/3.1/manual/query_timestamps/ > > <https://docs.datastax.com/en/developer/java-driver/3.1/man > ual/query_timestamps/> > > > > We use our own NTP cluster to reduce clock drift as much as > > possible, but public NTP servers are good enough for most > > uses. https://www.instaclustr.com/blog/2015/11/05/apache-cassandra > -synchronization/ > > <https://www.instaclustr.com/blog/2015/11/05/apache-cassand > ra-synchronization/> > > > > Cheers, > > Justin > > > > On Thu, 9 Feb 2017 at 16:09 Kant Kodali > <mailto:k...@peernova.com>> wrote: > > > > How does Cassandra achieve Linearizability with “Last write > > wins” (conflict resolution methods based on time-of-day clocks) ? > > > > Relying on synchronized clocks are almost certainly > > non-linearizable, because clock timestamps cannot be guaranteed > > to be consistent with actual event ordering due to clock skew. > > isn't it? > > > > Thanks! > > > > -- > > > > Justin Cameron > > > > Senior
Re: How does cassandra achieve Linearizability?
imistic, especially if >>>>>>> a key is under contention. With regard to the “last write” part >>>>>>> you’re asking about - w/ LWT Cassandra provides the timestamp >>>>>>> and manages it as part of the ballot, and it always is >>>>>>> increasing. See >>>>>>> org.apache.cassandra.service.ClientState#getTimestampForPaxos. >>>>>>> From the code: >>>>>>> >>>>>>> * Returns a timestamp suitable for paxos given the timestamp of >>>>>>>the last known commit (or in progress update). >>>>>>> * Paxos ensures that the timestamp it uses for commits respects >>>>>>>the serial order of those commits. It does so >>>>>>> * by having each replica reject any proposal whose timestamp is >>>>>>>not strictly greater than the last proposal it >>>>>>> * accepted. So in practice, which timestamp we use for a given >>>>>>>proposal doesn't affect correctness but it does >>>>>>> * affect the chance of making progress (if we pick a timestamp >>>>>>>lower than what has been proposed before, our >>>>>>> * new proposal will just get rejected). >>>>>>> >>>>>>> Effectively paxos removes the ability to use custom timestamps >>>>>>> and addresses clock variance by rejecting ballots with >>>>>>> timestamps less than what was last seen. You can learn more by >>>>>>> reading through the other comments and code in that file. >>>>>>> >>>>>>> Last write wins is a free for all that guarantees you *nothing* >>>>>>> except the timestamp is used as a tiebreaker. Here we >>>>>>> acknowledge things like the speed of light as being a real >>>>>>> problem that isn’t going away anytime soon. This problem is >>>>>>> sometimes addressed with event sourcing rather than mutating in >>>>>>> place. >>>>>>> >>>>>>> Hope this helps. >>>>>>> >>>>>>> >>>>>>> Jon >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> On Feb 9, 2017, at 5:21 PM, Kant Kodali >>>>>>>> wrote: >>>>>>>> >>>>>>>> @Justin I read this article >>>>>>>> http://www.datastax.com/dev/blog/lightweight-transactions-in-cassandra-2-0. >>>>>>>> And it clearly says Linearizable consistency can be achieved >>>>>>>> with LWT's. so should I assume the Linearizability in the >>>>>>>> context of the above article is possible with LWT's and >>>>>>>> synchronization of clocks through ntpd ? because LWT's also >>>>>>>> follow Last Write Wins. isn't it? Also another question does >>>>>>>> most of the production clusters do setup ntpd? If so what is >>>>>>>> the time it takes to sync? any idea >>>>>>>> >>>>>>>> @Micheal Schuler Are you referring to something like true time >>>>>>>> as in >>>>>>>> https://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf? >>>>>>>> Actually I never heard of setting up GPS modules and how that >>>>>>>> can be helpful. Let me research on that but good point. >>>>>>>> >>>>>>>> On Thu, Feb 9, 2017 at 5:09 PM, Michael Shuler >>>>>>>> wrote: >>>>>>>>> If you require the best precision you can get, setting up a >>>>>>>>> pair of >>>>>>>>> stratum 1 ntpd masters in each data center location with a GPS >>>>>>>>> modules >>>>>>>>> is not terribly complex. Low latency and jitter on servers you >>>>>>>>> manage. >>>>>>>>> 140ms is a long way away network-wise, and I would suggest >>>>>>>>> that was a >>>>>>>>> poor choice of upstream (probably stratum 2 or 3) source. >>>>>>>>> >>>>>>>&g
Re: How does cassandra achieve Linearizability?
Last write wins is a free for all that guarantees you *nothing* except >> the timestamp is used as a tiebreaker. Here we acknowledge things like the >> speed of light as being a real problem that isn’t going away anytime soon. >> This problem is sometimes addressed with event sourcing rather than >> mutating in place. >> >> Hope this helps. >> >> >> Jon >> >> >> >> >> On Feb 9, 2017, at 5:21 PM, Kant Kodali wrote: >> >> @Justin I read this article http://www.datastax.com/dev/bl >> og/lightweight-transactions-in-cassandra-2-0. And it clearly says >> Linearizable consistency can be achieved with LWT's. so should I assume >> the Linearizability in the context of the above article is possible with >> LWT's and synchronization of clocks through ntpd ? because LWT's also >> follow Last Write Wins. isn't it? Also another question does most of the >> production clusters do setup ntpd? If so what is the time it takes to sync? >> any idea >> >> @Micheal Schuler Are you referring to something like true time as in >> https://static.googleusercontent.com/media/research.google.c >> om/en//archive/spanner-osdi2012.pdf? Actually I never heard of setting >> up GPS modules and how that can be helpful. Let me research on that but >> good point. >> >> On Thu, Feb 9, 2017 at 5:09 PM, Michael Shuler >> wrote: >> >> If you require the best precision you can get, setting up a pair of >> stratum 1 ntpd masters in each data center location with a GPS modules >> is not terribly complex. Low latency and jitter on servers you manage. >> 140ms is a long way away network-wise, and I would suggest that was a >> poor choice of upstream (probably stratum 2 or 3) source. >> >> As Jonathan mentioned, there's no guarantee from Cassandra, but if you >> need as close as you can get, you'll probably need to do it yourself. >> >> (I run several stratum 2 ntpd servers for pool.ntp.org) >> >> -- >> Kind regards, >> Michael >> >> On 02/09/2017 06:47 PM, Kant Kodali wrote: >> > Hi Justin, >> > >> > There are bunch of issues w.r.t to synchronization of clocks when we >> > used ntpd. Also the time it took to sync the clocks was approx 140ms >> > (don't quote me on it though because it is reported by our devops :) >> > >> > we have multiple clients (for example bunch of micro services are >> > reading from Cassandra) I am not sure how one can achieve >> > Linearizability by setting timestamps on the clients ? since there is no >> > total ordering across multiple clients. >> > >> > Thanks! >> > >> > >> > On Thu, Feb 9, 2017 at 4:16 PM, Justin Cameron > > <mailto:jus...@instaclustr.com>> wrote: >> > >> > Hi Kant, >> > >> > Clock synchronization is important - you should ensure that ntpd is >> > properly configured on all nodes. If your particular use case is >> > especially sensitive to out-of-order mutations it is possible to set >> > timestamps on the client side using the >> > drivers. https://docs.datastax.com/en/d >> eveloper/java-driver/3.1/manual/query_timestamps/ >> > <https://docs.datastax.com/en/developer/java-driver/3.1/man >> ual/query_timestamps/> >> > >> > We use our own NTP cluster to reduce clock drift as much as >> > possible, but public NTP servers are good enough for most >> > uses. https://www.instaclustr.com/blog/2015/11/05/apache-cassandra >> -synchronization/ >> > <https://www.instaclustr.com/blog/2015/11/05/apache-cassand >> ra-synchronization/> >> > >> > Cheers, >> > Justin >> > >> > On Thu, 9 Feb 2017 at 16:09 Kant Kodali > > <mailto:k...@peernova.com>> wrote: >> > >> > How does Cassandra achieve Linearizability with “Last write >> > wins” (conflict resolution methods based on time-of-day clocks) >> ? >> > >> > Relying on synchronized clocks are almost certainly >> > non-linearizable, because clock timestamps cannot be guaranteed >> > to be consistent with actual event ordering due to clock skew. >> > isn't it? >> > >> > Thanks! >> > >> > -- >> > >> > Justin Cameron >> > >> > Senior Software Engineer | Instaclustr >> > >> > >> > >> > >> > This email has been sent on behalf of Instaclustr Pty Ltd >> > (Australia) and Instaclustr Inc (USA). >> > >> > This email and any attachments may contain confidential and legally >> > privileged information. If you are not the intended recipient, do >> > not copy or disclose its content, but please reply to this email >> > immediately and highlight the error to the sender and then >> > immediately delete the message. >> > >> > >> >> >> >> >> >> >> >> >> -- >> Benjamin Roth >> Prokurist >> >> Jaumo GmbH · www.jaumo.com >> Wehrstraße 46 · 73035 Göppingen · Germany >> Phone +49 7161 304880-6 <+49%207161%203048806> · Fax +49 7161 304880-1 >> <+49%207161%203048801> >> AG Ulm · HRB 731058 · Managing Director: Jens Kammerer >> >> >> >
Re: How does cassandra achieve Linearizability?
estion does most of the > production clusters do setup ntpd? If so what is the time it takes to sync? > any idea > > @Micheal Schuler Are you referring to something like true time as in > https://static.googleusercontent.com/media/research.google.c > om/en//archive/spanner-osdi2012.pdf? Actually I never heard of setting > up GPS modules and how that can be helpful. Let me research on that but > good point. > > On Thu, Feb 9, 2017 at 5:09 PM, Michael Shuler > wrote: > > If you require the best precision you can get, setting up a pair of > stratum 1 ntpd masters in each data center location with a GPS modules > is not terribly complex. Low latency and jitter on servers you manage. > 140ms is a long way away network-wise, and I would suggest that was a > poor choice of upstream (probably stratum 2 or 3) source. > > As Jonathan mentioned, there's no guarantee from Cassandra, but if you > need as close as you can get, you'll probably need to do it yourself. > > (I run several stratum 2 ntpd servers for pool.ntp.org) > > -- > Kind regards, > Michael > > On 02/09/2017 06:47 PM, Kant Kodali wrote: > > Hi Justin, > > > > There are bunch of issues w.r.t to synchronization of clocks when we > > used ntpd. Also the time it took to sync the clocks was approx 140ms > > (don't quote me on it though because it is reported by our devops :) > > > > we have multiple clients (for example bunch of micro services are > > reading from Cassandra) I am not sure how one can achieve > > Linearizability by setting timestamps on the clients ? since there is no > > total ordering across multiple clients. > > > > Thanks! > > > > > > On Thu, Feb 9, 2017 at 4:16 PM, Justin Cameron > <mailto:jus...@instaclustr.com>> wrote: > > > > Hi Kant, > > > > Clock synchronization is important - you should ensure that ntpd is > > properly configured on all nodes. If your particular use case is > > especially sensitive to out-of-order mutations it is possible to set > > timestamps on the client side using the > > drivers. https://docs.datastax.com/en/d > eveloper/java-driver/3.1/manual/query_timestamps/ > > <https://docs.datastax.com/en/developer/java-driver/3.1/man > ual/query_timestamps/> > > > > We use our own NTP cluster to reduce clock drift as much as > > possible, but public NTP servers are good enough for most > > uses. https://www.instaclustr.com/blog/2015/11/05/apache-cassandra > -synchronization/ > > <https://www.instaclustr.com/blog/2015/11/05/apache-cassand > ra-synchronization/> > > > > Cheers, > > Justin > > > > On Thu, 9 Feb 2017 at 16:09 Kant Kodali > <mailto:k...@peernova.com>> wrote: > > > > How does Cassandra achieve Linearizability with “Last write > > wins” (conflict resolution methods based on time-of-day clocks) ? > > > > Relying on synchronized clocks are almost certainly > > non-linearizable, because clock timestamps cannot be guaranteed > > to be consistent with actual event ordering due to clock skew. > > isn't it? > > > > Thanks! > > > > -- > > > > Justin Cameron > > > > Senior Software Engineer | Instaclustr > > > > > > > > > > This email has been sent on behalf of Instaclustr Pty Ltd > > (Australia) and Instaclustr Inc (USA). > > > > This email and any attachments may contain confidential and legally > > privileged information. If you are not the intended recipient, do > > not copy or disclose its content, but please reply to this email > > immediately and highlight the error to the sender and then > > immediately delete the message. > > > > > > > > > > > > > -- > Benjamin Roth > Prokurist > > Jaumo GmbH · www.jaumo.com > Wehrstraße 46 · 73035 Göppingen · Germany > Phone +49 7161 304880-6 <+49%207161%203048806> · Fax +49 7161 304880-1 > <+49%207161%203048801> > AG Ulm · HRB 731058 · Managing Director: Jens Kammerer > > >
Re: How does cassandra achieve Linearizability?
be achieved with >>>>> LWT's. so should I assume the Linearizability in the context of >>>>> the above article is possible with LWT's and synchronization of >>>>> clocks through ntpd ? because LWT's also follow Last Write Wins. >>>>> isn't it? Also another question does most of the production >>>>> clusters do setup ntpd? If so what is the time it takes to sync? >>>>> any idea >>>>> >>>>> @Micheal Schuler Are you referring to something like true time as >>>>> in >>>>> https://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf? >>>>> Actually I never heard of setting up GPS modules and how that can >>>>> be helpful. Let me research on that but good point. >>>>> >>>>> On Thu, Feb 9, 2017 at 5:09 PM, Michael Shuler >>>>> wrote: >>>>>> If you require the best precision you can get, setting up a >>>>>> pair of >>>>>> stratum 1 ntpd masters in each data center location with a GPS >>>>>> modules >>>>>> is not terribly complex. Low latency and jitter on servers you >>>>>> manage. >>>>>> 140ms is a long way away network-wise, and I would suggest that >>>>>> was a >>>>>> poor choice of upstream (probably stratum 2 or 3) source. >>>>>> >>>>>> As Jonathan mentioned, there's no guarantee from Cassandra, but >>>>>> if you >>>>>> need as close as you can get, you'll probably need to do it >>>>>> yourself. >>>>>> >>>>>> (I run several stratum 2 ntpd servers for pool.ntp.org[2]) >>>>>> >>>>>> -- >>>>>> Kind regards, Michael >>>>>> >>>>>> On 02/09/2017 06:47 PM, Kant Kodali wrote: >>>>>> > Hi Justin, >>>>>> > >>>>>> > There are bunch of issues w.r.t to synchronization of clocks >>>>>> > when we used ntpd. Also the time it took to sync the clocks >>>>>> > was approx 140ms (don't quote me on it though because it is >>>>>> > reported by our devops :) >>>>>> > >>>>>> > we have multiple clients (for example bunch of micro services >>>>>> > are reading from Cassandra) I am not sure how one can achieve >>>>>> > Linearizability by setting timestamps on the clients ? since >>>>>> > there is no total ordering across multiple clients. >>>>>> > >>>>>> > Thanks! >>>>>> > >>>>>> > >>>>>> > On Thu, Feb 9, 2017 at 4:16 PM, Justin Cameron >>>>>> > >>>>> > <mailto:jus...@instaclustr.com>> wrote: >>>>>> > >>>>>> > Hi Kant, >>>>>> > >>>>>> > Clock synchronization is important - you should ensure >>>>>> > that ntpd is properly configured on all nodes. If your >>>>>> > particular use case is especially sensitive to out-of- >>>>>> > order mutations it is possible to set timestamps on the >>>>>> > client side using the drivers. >>>>>> > >>>>>> https://docs.datastax.com/en/developer/java-driver/3.1/manual/query_timestamps/ >>>>>> > >>>>>> <https://docs.datastax.com/en/developer/java-driver/3.1/manual/query_timestamps/> >>>>>> > >>>>>> > We use our own NTP cluster to reduce clock drift as much >>>>>> > as possible, but public NTP servers are good enough for >>>>>> > most uses. >>>>>> > >>>>>> https://www.instaclustr.com/blog/2015/11/05/apache-cassandra-synchronization/ >>>>>> > >>>>>> <https://www.instaclustr.com/blog/2015/11/05/apache-cassandra-synchronization/> >>>>>> > >>>>>> > Cheers, Justin >>>>>> > >>>>>> > On Thu, 9 Feb 2017 at 16:09 Kant Kodali >>>>> > <mailto:k...@peernova.com>> wrote: >>>>>> > >>>>>> > How does Cassandra achieve Linearizability with “Last >>>>>> > write wins” (conflict resolution methods based on time-of- >>>>>> > day clocks) ? >>>>>> > >>>>>> > Relying on synchronized clocks are almost certainly >>>>>> > non-linearizable, because clock timestamps cannot be >>>>>> > guaranteed to be consistent with actual event ordering >>>>>> > due to clock skew. isn't it? >>>>>> > >>>>>> > Thanks! >>>>>> > >>>>>> > -- >>>>>> > >>>>>> > Justin Cameron >>>>>> > >>>>>> > Senior Software Engineer | Instaclustr >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > This email has been sent on behalf of Instaclustr Pty Ltd >>>>>> > (Australia) and Instaclustr Inc (USA). >>>>>> > >>>>>> > This email and any attachments may contain confidential >>>>>> > and legally >>>>>> > privileged information. If you are not the intended >>>>>> > recipient, do >>>>>> > not copy or disclose its content, but please reply to this >>>>>> > email >>>>>> > immediately and highlight the error to the sender and then >>>>>> > immediately delete the message. >>>>>> > >>>>>> > >>>>>> >>>>> >>>> >>> >> > > > > -- > Benjamin Roth > Prokurist > > Jaumo GmbH · www.jaumo.com > Wehrstraße 46 · 73035 Göppingen · Germany > Phone +49 7161 304880-6 · Fax +49 7161 304880-1 > AG Ulm · HRB 731058 · Managing Director: Jens Kammerer Links: 1. https://issues.apache.org/jira/browse/CASSANDRA-6246?jql=text%20~%20%22epaxos%22 2. http://pool.ntp.org/
Re: How does cassandra achieve Linearizability?
>>>> 140ms is a long way away network-wise, and I would suggest that was a >>>> poor choice of upstream (probably stratum 2 or 3) source. >>>> >>>> As Jonathan mentioned, there's no guarantee from Cassandra, but if you >>>> need as close as you can get, you'll probably need to do it yourself. >>>> >>>> (I run several stratum 2 ntpd servers for pool.ntp.org) >>>> >>>> -- >>>> Kind regards, >>>> Michael >>>> >>>> On 02/09/2017 06:47 PM, Kant Kodali wrote: >>>> > Hi Justin, >>>> > >>>> > There are bunch of issues w.r.t to synchronization of clocks when we >>>> > used ntpd. Also the time it took to sync the clocks was approx 140ms >>>> > (don't quote me on it though because it is reported by our devops :) >>>> > >>>> > we have multiple clients (for example bunch of micro services are >>>> > reading from Cassandra) I am not sure how one can achieve >>>> > Linearizability by setting timestamps on the clients ? since there is >>>> no >>>> > total ordering across multiple clients. >>>> > >>>> > Thanks! >>>> > >>>> > >>>> > On Thu, Feb 9, 2017 at 4:16 PM, Justin Cameron < >>>> jus...@instaclustr.com >>>> > <mailto:jus...@instaclustr.com>> wrote: >>>> > >>>> > Hi Kant, >>>> > >>>> > Clock synchronization is important - you should ensure that ntpd >>>> is >>>> > properly configured on all nodes. If your particular use case is >>>> > especially sensitive to out-of-order mutations it is possible to >>>> set >>>> > timestamps on the client side using the >>>> > drivers. https://docs.datastax.com/en/d >>>> eveloper/java-driver/3.1/manual/query_timestamps/ >>>> > <https://docs.datastax.com/en/developer/java-driver/3.1/man >>>> ual/query_timestamps/> >>>> > >>>> > We use our own NTP cluster to reduce clock drift as much as >>>> > possible, but public NTP servers are good enough for most >>>> > uses. https://www.instaclustr.com/bl >>>> og/2015/11/05/apache-cassandra-synchronization/ >>>> > <https://www.instaclustr.com/blog/2015/11/05/apache-cassand >>>> ra-synchronization/> >>>> > >>>> > Cheers, >>>> > Justin >>>> > >>>> > On Thu, 9 Feb 2017 at 16:09 Kant Kodali >>> > <mailto:k...@peernova.com>> wrote: >>>> > >>>> > How does Cassandra achieve Linearizability with “Last write >>>> > wins” (conflict resolution methods based on time-of-day >>>> clocks) ? >>>> > >>>> > Relying on synchronized clocks are almost certainly >>>> > non-linearizable, because clock timestamps cannot be >>>> guaranteed >>>> > to be consistent with actual event ordering due to clock skew. >>>> > isn't it? >>>> > >>>> > Thanks! >>>> > >>>> > -- >>>> > >>>> > Justin Cameron >>>> > >>>> > Senior Software Engineer | Instaclustr >>>> > >>>> > >>>> > >>>> > >>>> > This email has been sent on behalf of Instaclustr Pty Ltd >>>> > (Australia) and Instaclustr Inc (USA). >>>> > >>>> > This email and any attachments may contain confidential and >>>> legally >>>> > privileged information. If you are not the intended recipient, do >>>> > not copy or disclose its content, but please reply to this email >>>> > immediately and highlight the error to the sender and then >>>> > immediately delete the message. >>>> > >>>> > >>>> >>>> >>> >>> >> > -- Benjamin Roth Prokurist Jaumo GmbH · www.jaumo.com Wehrstraße 46 · 73035 Göppingen · Germany Phone +49 7161 304880-6 · Fax +49 7161 304880-1 AG Ulm · HRB 731058 · Managing Director: Jens Kammerer
Re: How does cassandra achieve Linearizability?
lients (for example bunch of micro services are >>> > reading from Cassandra) I am not sure how one can achieve >>> > Linearizability by setting timestamps on the clients ? since there is >>> no >>> > total ordering across multiple clients. >>> > >>> > Thanks! >>> > >>> > >>> > On Thu, Feb 9, 2017 at 4:16 PM, Justin Cameron >> > <mailto:jus...@instaclustr.com>> wrote: >>> > >>> > Hi Kant, >>> > >>> > Clock synchronization is important - you should ensure that ntpd is >>> > properly configured on all nodes. If your particular use case is >>> > especially sensitive to out-of-order mutations it is possible to >>> set >>> > timestamps on the client side using the >>> > drivers. https://docs.datastax.com/en/d >>> eveloper/java-driver/3.1/manual/query_timestamps/ >>> > <https://docs.datastax.com/en/developer/java-driver/3.1/man >>> ual/query_timestamps/> >>> > >>> > We use our own NTP cluster to reduce clock drift as much as >>> > possible, but public NTP servers are good enough for most >>> > uses. https://www.instaclustr.com/blog/2015/11/05/apache-cassandra >>> -synchronization/ >>> > <https://www.instaclustr.com/blog/2015/11/05/apache-cassand >>> ra-synchronization/> >>> > >>> > Cheers, >>> > Justin >>> > >>> > On Thu, 9 Feb 2017 at 16:09 Kant Kodali >> > <mailto:k...@peernova.com>> wrote: >>> > >>> > How does Cassandra achieve Linearizability with “Last write >>> > wins” (conflict resolution methods based on time-of-day >>> clocks) ? >>> > >>> > Relying on synchronized clocks are almost certainly >>> > non-linearizable, because clock timestamps cannot be guaranteed >>> > to be consistent with actual event ordering due to clock skew. >>> > isn't it? >>> > >>> > Thanks! >>> > >>> > -- >>> > >>> > Justin Cameron >>> > >>> > Senior Software Engineer | Instaclustr >>> > >>> > >>> > >>> > >>> > This email has been sent on behalf of Instaclustr Pty Ltd >>> > (Australia) and Instaclustr Inc (USA). >>> > >>> > This email and any attachments may contain confidential and legally >>> > privileged information. If you are not the intended recipient, do >>> > not copy or disclose its content, but please reply to this email >>> > immediately and highlight the error to the sender and then >>> > immediately delete the message. >>> > >>> > >>> >>> >> >> >
Re: How does cassandra achieve Linearizability?
nodes. If your particular use case is >> > especially sensitive to out-of-order mutations it is possible to set >> > timestamps on the client side using the >> > drivers. https://docs.datastax.com/en/d >> eveloper/java-driver/3.1/manual/query_timestamps/ >> > <https://docs.datastax.com/en/developer/java-driver/3.1/m >> anual/query_timestamps/> >> > >> > We use our own NTP cluster to reduce clock drift as much as >> > possible, but public NTP servers are good enough for most >> > uses. https://www.instaclustr.com/blog/2015/11/05/apache-cassandra >> -synchronization/ >> > <https://www.instaclustr.com/blog/2015/11/05/apache-cassand >> ra-synchronization/> >> > >> > Cheers, >> > Justin >> > >> > On Thu, 9 Feb 2017 at 16:09 Kant Kodali > > <mailto:k...@peernova.com>> wrote: >> > >> > How does Cassandra achieve Linearizability with “Last write >> > wins” (conflict resolution methods based on time-of-day clocks) >> ? >> > >> > Relying on synchronized clocks are almost certainly >> > non-linearizable, because clock timestamps cannot be guaranteed >> > to be consistent with actual event ordering due to clock skew. >> > isn't it? >> > >> > Thanks! >> > >> > -- >> > >> > Justin Cameron >> > >> > Senior Software Engineer | Instaclustr >> > >> > >> > >> > >> > This email has been sent on behalf of Instaclustr Pty Ltd >> > (Australia) and Instaclustr Inc (USA). >> > >> > This email and any attachments may contain confidential and legally >> > privileged information. If you are not the intended recipient, do >> > not copy or disclose its content, but please reply to this email >> > immediately and highlight the error to the sender and then >> > immediately delete the message. >> > >> > >> >> > >
Re: How does cassandra achieve Linearizability?
> > https://docs.datastax.com/en/developer/java-driver/3.1/manual/query_timestamps/ > > > > <https://docs.datastax.com/en/developer/java-driver/3.1/manual/query_timestamps/> > > > > <https://docs.datastax.com/en/developer/java-driver/3.1/manual/query_timestamps/ > > > > <https://docs.datastax.com/en/developer/java-driver/3.1/manual/query_timestamps/>> > > > > We use our own NTP cluster to reduce clock drift as much as > > possible, but public NTP servers are good enough for most > > uses. > > https://www.instaclustr.com/blog/2015/11/05/apache-cassandra-synchronization/ > > > > <https://www.instaclustr.com/blog/2015/11/05/apache-cassandra-synchronization/> > > > > <https://www.instaclustr.com/blog/2015/11/05/apache-cassandra-synchronization/ > > > > <https://www.instaclustr.com/blog/2015/11/05/apache-cassandra-synchronization/>> > > > > Cheers, > > Justin > > > > On Thu, 9 Feb 2017 at 16:09 Kant Kodali > <mailto:k...@peernova.com> > > <mailto:k...@peernova.com <mailto:k...@peernova.com>>> wrote: > > > > How does Cassandra achieve Linearizability with “Last write > > wins” (conflict resolution methods based on time-of-day clocks) ? > > > > Relying on synchronized clocks are almost certainly > > non-linearizable, because clock timestamps cannot be guaranteed > > to be consistent with actual event ordering due to clock skew. > > isn't it? > > > > Thanks! > > > > -- > > > > Justin Cameron > > > > Senior Software Engineer | Instaclustr > > > > > > > > > > This email has been sent on behalf of Instaclustr Pty Ltd > > (Australia) and Instaclustr Inc (USA). > > > > This email and any attachments may contain confidential and legally > > privileged information. If you are not the intended recipient, do > > not copy or disclose its content, but please reply to this email > > immediately and highlight the error to the sender and then > > immediately delete the message. > > > > > >
Re: How does cassandra achieve Linearizability?
On 02/09/2017 07:21 PM, Kant Kodali wrote: > @Justin I read this article > http://www.datastax.com/dev/blog/lightweight-transactions-in-cassandra-2-0. > And it clearly says Linearizable consistency can be achieved with LWT's. > so should I assume the Linearizability in the context of the above > article is possible with LWT's and synchronization of clocks through > ntpd ? because LWT's also follow Last Write Wins. isn't it? Also another > question does most of the production clusters do setup ntpd? If so what > is the time it takes to sync? any idea I'll let the others talk more intimately about LWT, but as for NTP, the client machines do take some time to incrementally settle time adjustments to meet up with the upstreams - they don't just jump time. > @Micheal Schuler Are you referring to something like true time as in > https://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf? > > Actually I never heard of setting up GPS modules and how that can be > helpful. Let me research on that but good point. Nah, I'm talking much simpler. For instance you could do this with a raspberry pi: http://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html -- Michael > On Thu, Feb 9, 2017 at 5:09 PM, Michael Shuler <mailto:mich...@pbandjelly.org>> wrote: > > If you require the best precision you can get, setting up a pair of > stratum 1 ntpd masters in each data center location with a GPS modules > is not terribly complex. Low latency and jitter on servers you manage. > 140ms is a long way away network-wise, and I would suggest that was a > poor choice of upstream (probably stratum 2 or 3) source. > > As Jonathan mentioned, there's no guarantee from Cassandra, but if you > need as close as you can get, you'll probably need to do it yourself. > > (I run several stratum 2 ntpd servers for pool.ntp.org > <http://pool.ntp.org>) > > -- > Kind regards, > Michael > > On 02/09/2017 06:47 PM, Kant Kodali wrote: > > Hi Justin, > > > > There are bunch of issues w.r.t to synchronization of clocks when we > > used ntpd. Also the time it took to sync the clocks was approx 140ms > > (don't quote me on it though because it is reported by our devops :) > > > > we have multiple clients (for example bunch of micro services are > > reading from Cassandra) I am not sure how one can achieve > > Linearizability by setting timestamps on the clients ? since there is no > > total ordering across multiple clients. > > > > Thanks! > > > > > > On Thu, Feb 9, 2017 at 4:16 PM, Justin Cameron <mailto:jus...@instaclustr.com> > > <mailto:jus...@instaclustr.com <mailto:jus...@instaclustr.com>>> wrote: > > > > Hi Kant, > > > > Clock synchronization is important - you should ensure that ntpd is > > properly configured on all nodes. If your particular use case is > > especially sensitive to out-of-order mutations it is possible to set > > timestamps on the client side using the > > drivers. > https://docs.datastax.com/en/developer/java-driver/3.1/manual/query_timestamps/ > > <https://docs.datastax.com/en/developer/java-driver/3.1/manual/query_timestamps/> > > > <https://docs.datastax.com/en/developer/java-driver/3.1/manual/query_timestamps/ > > <https://docs.datastax.com/en/developer/java-driver/3.1/manual/query_timestamps/>> > > > > We use our own NTP cluster to reduce clock drift as much as > > possible, but public NTP servers are good enough for most > > uses. > https://www.instaclustr.com/blog/2015/11/05/apache-cassandra-synchronization/ > > <https://www.instaclustr.com/blog/2015/11/05/apache-cassandra-synchronization/> > > > <https://www.instaclustr.com/blog/2015/11/05/apache-cassandra-synchronization/ > > <https://www.instaclustr.com/blog/2015/11/05/apache-cassandra-synchronization/>> > > > > Cheers, > > Justin > > > > On Thu, 9 Feb 2017 at 16:09 Kant Kodali <mailto:k...@peernova.com> > > <mailto:k...@peernova.com <mailto:k...@peernova.com>>> wrote: > > > > How does Cassandra achieve Linearizability with “Last write > > wins” (conflict resolution methods based on time-of-day clocks) > ? > > > > Relying on synchronized clocks are almost certainly > > non
Re: How does cassandra achieve Linearizability?
@Justin I read this article http://www.datastax.com/dev/blog/lightweight-transactions-in-cassandra-2-0. And it clearly says Linearizable consistency can be achieved with LWT's. so should I assume the Linearizability in the context of the above article is possible with LWT's and synchronization of clocks through ntpd ? because LWT's also follow Last Write Wins. isn't it? Also another question does most of the production clusters do setup ntpd? If so what is the time it takes to sync? any idea @Micheal Schuler Are you referring to something like true time as in https://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf? Actually I never heard of setting up GPS modules and how that can be helpful. Let me research on that but good point. On Thu, Feb 9, 2017 at 5:09 PM, Michael Shuler wrote: > If you require the best precision you can get, setting up a pair of > stratum 1 ntpd masters in each data center location with a GPS modules > is not terribly complex. Low latency and jitter on servers you manage. > 140ms is a long way away network-wise, and I would suggest that was a > poor choice of upstream (probably stratum 2 or 3) source. > > As Jonathan mentioned, there's no guarantee from Cassandra, but if you > need as close as you can get, you'll probably need to do it yourself. > > (I run several stratum 2 ntpd servers for pool.ntp.org) > > -- > Kind regards, > Michael > > On 02/09/2017 06:47 PM, Kant Kodali wrote: > > Hi Justin, > > > > There are bunch of issues w.r.t to synchronization of clocks when we > > used ntpd. Also the time it took to sync the clocks was approx 140ms > > (don't quote me on it though because it is reported by our devops :) > > > > we have multiple clients (for example bunch of micro services are > > reading from Cassandra) I am not sure how one can achieve > > Linearizability by setting timestamps on the clients ? since there is no > > total ordering across multiple clients. > > > > Thanks! > > > > > > On Thu, Feb 9, 2017 at 4:16 PM, Justin Cameron > <mailto:jus...@instaclustr.com>> wrote: > > > > Hi Kant, > > > > Clock synchronization is important - you should ensure that ntpd is > > properly configured on all nodes. If your particular use case is > > especially sensitive to out-of-order mutations it is possible to set > > timestamps on the client side using the > > drivers. https://docs.datastax.com/en/developer/java-driver/3.1/ > manual/query_timestamps/ > > <https://docs.datastax.com/en/developer/java-driver/3.1/ > manual/query_timestamps/> > > > > We use our own NTP cluster to reduce clock drift as much as > > possible, but public NTP servers are good enough for most > > uses. https://www.instaclustr.com/blog/2015/11/05/apache- > cassandra-synchronization/ > > <https://www.instaclustr.com/blog/2015/11/05/apache- > cassandra-synchronization/> > > > > Cheers, > > Justin > > > > On Thu, 9 Feb 2017 at 16:09 Kant Kodali > <mailto:k...@peernova.com>> wrote: > > > > How does Cassandra achieve Linearizability with “Last write > > wins” (conflict resolution methods based on time-of-day clocks) ? > > > > Relying on synchronized clocks are almost certainly > > non-linearizable, because clock timestamps cannot be guaranteed > > to be consistent with actual event ordering due to clock skew. > > isn't it? > > > > Thanks! > > > > -- > > > > Justin Cameron > > > > Senior Software Engineer | Instaclustr > > > > > > > > > > This email has been sent on behalf of Instaclustr Pty Ltd > > (Australia) and Instaclustr Inc (USA). > > > > This email and any attachments may contain confidential and legally > > privileged information. If you are not the intended recipient, do > > not copy or disclose its content, but please reply to this email > > immediately and highlight the error to the sender and then > > immediately delete the message. > > > > > >
Re: How does cassandra achieve Linearizability?
If you require the best precision you can get, setting up a pair of stratum 1 ntpd masters in each data center location with a GPS modules is not terribly complex. Low latency and jitter on servers you manage. 140ms is a long way away network-wise, and I would suggest that was a poor choice of upstream (probably stratum 2 or 3) source. As Jonathan mentioned, there's no guarantee from Cassandra, but if you need as close as you can get, you'll probably need to do it yourself. (I run several stratum 2 ntpd servers for pool.ntp.org) -- Kind regards, Michael On 02/09/2017 06:47 PM, Kant Kodali wrote: > Hi Justin, > > There are bunch of issues w.r.t to synchronization of clocks when we > used ntpd. Also the time it took to sync the clocks was approx 140ms > (don't quote me on it though because it is reported by our devops :) > > we have multiple clients (for example bunch of micro services are > reading from Cassandra) I am not sure how one can achieve > Linearizability by setting timestamps on the clients ? since there is no > total ordering across multiple clients. > > Thanks! > > > On Thu, Feb 9, 2017 at 4:16 PM, Justin Cameron <mailto:jus...@instaclustr.com>> wrote: > > Hi Kant, > > Clock synchronization is important - you should ensure that ntpd is > properly configured on all nodes. If your particular use case is > especially sensitive to out-of-order mutations it is possible to set > timestamps on the client side using the > drivers. > https://docs.datastax.com/en/developer/java-driver/3.1/manual/query_timestamps/ > > <https://docs.datastax.com/en/developer/java-driver/3.1/manual/query_timestamps/> > > We use our own NTP cluster to reduce clock drift as much as > possible, but public NTP servers are good enough for most > uses. > https://www.instaclustr.com/blog/2015/11/05/apache-cassandra-synchronization/ > > <https://www.instaclustr.com/blog/2015/11/05/apache-cassandra-synchronization/> > > Cheers, > Justin > > On Thu, 9 Feb 2017 at 16:09 Kant Kodali <mailto:k...@peernova.com>> wrote: > > How does Cassandra achieve Linearizability with “Last write > wins” (conflict resolution methods based on time-of-day clocks) ? > > Relying on synchronized clocks are almost certainly > non-linearizable, because clock timestamps cannot be guaranteed > to be consistent with actual event ordering due to clock skew. > isn't it? > > Thanks! > > -- > > Justin Cameron > > Senior Software Engineer | Instaclustr > > > > > This email has been sent on behalf of Instaclustr Pty Ltd > (Australia) and Instaclustr Inc (USA). > > This email and any attachments may contain confidential and legally > privileged information. If you are not the intended recipient, do > not copy or disclose its content, but please reply to this email > immediately and highlight the error to the sender and then > immediately delete the message. > >
Re: How does cassandra achieve Linearizability?
I think the answer to that question will depend on your specific use case and requirements. If you're only doing a small number of updates but need to be sure they are applied in order you may be able to use lightweight transactions (keep in mind there's a performance hit here, so it's not an answer for high-volume mutations). For high-volume updates you could look at using an append-only time-series style data model, using a default TTL to drop old data. If your data isn't time-series in nature and has a high-volume of updates then you really just need to make sure either your clients or Cassandra nodes (preferably both) are in sync. Justin On Thu, 9 Feb 2017 at 16:47 Kant Kodali wrote: > Hi Justin, > > There are bunch of issues w.r.t to synchronization of clocks when we used > ntpd. Also the time it took to sync the clocks was approx 140ms (don't > quote me on it though because it is reported by our devops :) > > we have multiple clients (for example bunch of micro services are reading > from Cassandra) I am not sure how one can achieve Linearizability by > setting timestamps on the clients ? since there is no total ordering across > multiple clients. > > Thanks! > > > On Thu, Feb 9, 2017 at 4:16 PM, Justin Cameron > wrote: > > Hi Kant, > > Clock synchronization is important - you should ensure that ntpd is > properly configured on all nodes. If your particular use case is especially > sensitive to out-of-order mutations it is possible to set timestamps on the > client side using the drivers. > https://docs.datastax.com/en/developer/java-driver/3.1/manual/query_timestamps/ > > We use our own NTP cluster to reduce clock drift as much as possible, but > public NTP servers are good enough for most uses. > https://www.instaclustr.com/blog/2015/11/05/apache-cassandra-synchronization/ > > Cheers, > Justin > > On Thu, 9 Feb 2017 at 16:09 Kant Kodali wrote: > > How does Cassandra achieve Linearizability with “Last write wins” > (conflict resolution methods based on time-of-day clocks) ? > > Relying on synchronized clocks are almost certainly non-linearizable, > because clock timestamps cannot be guaranteed to be consistent with actual > event ordering due to clock skew. isn't it? > > Thanks! > > -- > > Justin Cameron > > Senior Software Engineer | Instaclustr > > > > > This email has been sent on behalf of Instaclustr Pty Ltd (Australia) and > Instaclustr Inc (USA). > > This email and any attachments may contain confidential and legally > privileged information. If you are not the intended recipient, do not copy > or disclose its content, but please reply to this email immediately and > highlight the error to the sender and then immediately delete the message. > > > -- Justin Cameron Senior Software Engineer | Instaclustr This email has been sent on behalf of Instaclustr Pty Ltd (Australia) and Instaclustr Inc (USA). This email and any attachments may contain confidential and legally privileged information. If you are not the intended recipient, do not copy or disclose its content, but please reply to this email immediately and highlight the error to the sender and then immediately delete the message.
Re: How does cassandra achieve Linearizability?
It doesn't, nor does it claim to. On Thu, Feb 9, 2017 at 4:09 PM Kant Kodali wrote: > How does Cassandra achieve Linearizability with “Last write wins” > (conflict resolution methods based on time-of-day clocks) ? > > Relying on synchronized clocks are almost certainly non-linearizable, > because clock timestamps cannot be guaranteed to be consistent with actual > event ordering due to clock skew. isn't it? > > Thanks! >
Re: How does cassandra achieve Linearizability?
Hi Justin, There are bunch of issues w.r.t to synchronization of clocks when we used ntpd. Also the time it took to sync the clocks was approx 140ms (don't quote me on it though because it is reported by our devops :) we have multiple clients (for example bunch of micro services are reading from Cassandra) I am not sure how one can achieve Linearizability by setting timestamps on the clients ? since there is no total ordering across multiple clients. Thanks! On Thu, Feb 9, 2017 at 4:16 PM, Justin Cameron wrote: > Hi Kant, > > Clock synchronization is important - you should ensure that ntpd is > properly configured on all nodes. If your particular use case is especially > sensitive to out-of-order mutations it is possible to set timestamps on the > client side using the drivers. https://docs.datastax.com/en/developer/ > java-driver/3.1/manual/query_timestamps/ > > We use our own NTP cluster to reduce clock drift as much as possible, but > public NTP servers are good enough for most uses. https://www.instaclustr. > com/blog/2015/11/05/apache-cassandra-synchronization/ > > Cheers, > Justin > > On Thu, 9 Feb 2017 at 16:09 Kant Kodali wrote: > >> How does Cassandra achieve Linearizability with “Last write wins” >> (conflict resolution methods based on time-of-day clocks) ? >> >> Relying on synchronized clocks are almost certainly non-linearizable, >> because clock timestamps cannot be guaranteed to be consistent with actual >> event ordering due to clock skew. isn't it? >> >> Thanks! >> > -- > > Justin Cameron > > Senior Software Engineer | Instaclustr > > > > > This email has been sent on behalf of Instaclustr Pty Ltd (Australia) and > Instaclustr Inc (USA). > > This email and any attachments may contain confidential and legally > privileged information. If you are not the intended recipient, do not copy > or disclose its content, but please reply to this email immediately and > highlight the error to the sender and then immediately delete the message. > >
Re: How does cassandra achieve Linearizability?
Hi Kant, Clock synchronization is important - you should ensure that ntpd is properly configured on all nodes. If your particular use case is especially sensitive to out-of-order mutations it is possible to set timestamps on the client side using the drivers. https://docs.datastax.com/en/developer/java-driver/3.1/manual/query_timestamps/ We use our own NTP cluster to reduce clock drift as much as possible, but public NTP servers are good enough for most uses. https://www.instaclustr.com/blog/2015/11/05/apache-cassandra-synchronization/ Cheers, Justin On Thu, 9 Feb 2017 at 16:09 Kant Kodali wrote: > How does Cassandra achieve Linearizability with “Last write wins” > (conflict resolution methods based on time-of-day clocks) ? > > Relying on synchronized clocks are almost certainly non-linearizable, > because clock timestamps cannot be guaranteed to be consistent with actual > event ordering due to clock skew. isn't it? > > Thanks! > -- Justin Cameron Senior Software Engineer | Instaclustr This email has been sent on behalf of Instaclustr Pty Ltd (Australia) and Instaclustr Inc (USA). This email and any attachments may contain confidential and legally privileged information. If you are not the intended recipient, do not copy or disclose its content, but please reply to this email immediately and highlight the error to the sender and then immediately delete the message.
How does cassandra achieve Linearizability?
How does Cassandra achieve Linearizability with “Last write wins” (conflict resolution methods based on time-of-day clocks) ? Relying on synchronized clocks are almost certainly non-linearizable, because clock timestamps cannot be guaranteed to be consistent with actual event ordering due to clock skew. isn't it? Thanks!
