[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[ https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001924#comment-15001924 ] Stefan Podkowinski commented on CASSANDRA-9131: --- Since CASSANDRA-9649 {{state.getTimestamp()}} is now monotonically increasing for the VM instead of only for each connection. Operating this way should be pretty safe, as people should be well aware of the importance to keep clocks in sync across the cluster. Now in that situation I don't really feel comfortable with moving timestamp generation to the clients, as now potentially much more servers are affected and need to be monitored for clock skew. The chances will be much higher to have a single system introduce incorrect timestamp values that will corrupt your data. > Defining correct behavior during leap second insertion > -- > > Key: CASSANDRA-9131 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9131 > Project: Cassandra > Issue Type: Bug > Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue > Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux >Reporter: Jim Witschey >Priority: Minor > Fix For: 3.x > > > On Linux platforms, the insertion of a leap second breaks the monotonicity of > timestamps. This can make values appear to have been inserted into Cassandra > in a different order than they were. I want to know what behavior is expected > and desirable for inserts over this discontinuity. > From a timestamp perspective, an inserted leap second looks like a repeat of > the previous second: > {code} > $ while true ; do echo "`date +%s%N` `date -u`" ; sleep .5 ; done > 1435708798171327029 Tue Jun 30 23:59:58 UTC 2015 > 1435708798679392477 Tue Jun 30 23:59:58 UTC 2015 > 1435708799187550335 Tue Jun 30 23:59:59 UTC 2015 > 1435708799695670453 Tue Jun 30 23:59:59 UTC 2015 > 1435708799203902068 Tue Jun 30 23:59:59 UTC 2015 > 1435708799712168566 Tue Jun 30 23:59:59 UTC 2015 > 1435708800220473932 Wed Jul 1 00:00:00 UTC 2015 > 1435708800728908190 Wed Jul 1 00:00:00 UTC 2015 > 1435708801237611983 Wed Jul 1 00:00:01 UTC 2015 > 1435708801746251996 Wed Jul 1 00:00:01 UTC 2015 > {code} > Note that 23:59:59 repeats itself, and that the timestamps increase during > the first time through, then step back down to the beginning of the second > and increase again. > As a result, the timestamps on values inserted during these seconds will be > out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and > synced them to shortly before the leap second would be inserted. During the > insertion of the leap second, I ran a test with logic something like: > {code} > simple_insert = session.prepare( > 'INSERT INTO test (foo, bar) VALUES (?, ?);') > for i in itertools.count(): > # stop after midnight > now = datetime.utcnow() > last_midnight = now.replace(hour=0, minute=0, > second=0, microsecond=0) > seconds_since_midnight = (now - last_midnight).total_seconds() > if 5 <= seconds_since_midnight <= 15: > break > session.execute(simple_insert, [i, i]) > result = session.execute("SELECT bar, WRITETIME(bar) FROM test;") > {code} > EDIT: This behavior occurs with server-generated timestamps; in this > particular test, I set {{use_client_timestamp}} to {{False}}. > Under normal circumstances, the values and writetimes would increase > together, but when inserted over the leap second, they don't. These {{value, > writetime}} pairs are sorted by writetime: > {code} > (582, 1435708799285000) > (579, 1435708799339000) > (583, 1435708799593000) > (580, 1435708799643000) > (584, 1435708799897000) > (581, 1435708799958000) > {code} > The values were inserted in increasing order, but their writetimes are in a > different order because of the repeated second. During the first instance of > 23:59:59, the values 579, 580, and 581 were inserted at the beginning, > middle, and end of the second. During the leap second, which is also > 23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and > end of the second. However, since the two seconds are the same second, they > appear interleaved with respect to timestamps, as shown above. > So, should I consider this behavior correct? If not, how should Cassandra > correctly handle the discontinuity introduced by the insertion of a leap > second? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[ https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14527114#comment-14527114 ] Ryan McGuire commented on CASSANDRA-9131: - Assigned to [~slebresne] to provide protocol spec. > Defining correct behavior during leap second insertion > -- > > Key: CASSANDRA-9131 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9131 > Project: Cassandra > Issue Type: Bug > Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue > Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux >Reporter: Jim Witschey >Assignee: Sylvain Lebresne > > On Linux platforms, the insertion of a leap second breaks the monotonicity of > timestamps. This can make values appear to have been inserted into Cassandra > in a different order than they were. I want to know what behavior is expected > and desirable for inserts over this discontinuity. > From a timestamp perspective, an inserted leap second looks like a repeat of > the previous second: > {code} > $ while true ; do echo "`date +%s%N` `date -u`" ; sleep .5 ; done > 1435708798171327029 Tue Jun 30 23:59:58 UTC 2015 > 1435708798679392477 Tue Jun 30 23:59:58 UTC 2015 > 1435708799187550335 Tue Jun 30 23:59:59 UTC 2015 > 1435708799695670453 Tue Jun 30 23:59:59 UTC 2015 > 1435708799203902068 Tue Jun 30 23:59:59 UTC 2015 > 1435708799712168566 Tue Jun 30 23:59:59 UTC 2015 > 1435708800220473932 Wed Jul 1 00:00:00 UTC 2015 > 1435708800728908190 Wed Jul 1 00:00:00 UTC 2015 > 1435708801237611983 Wed Jul 1 00:00:01 UTC 2015 > 1435708801746251996 Wed Jul 1 00:00:01 UTC 2015 > {code} > Note that 23:59:59 repeats itself, and that the timestamps increase during > the first time through, then step back down to the beginning of the second > and increase again. > As a result, the timestamps on values inserted during these seconds will be > out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and > synced them to shortly before the leap second would be inserted. During the > insertion of the leap second, I ran a test with logic something like: > {code} > simple_insert = session.prepare( > 'INSERT INTO test (foo, bar) VALUES (?, ?);') > for i in itertools.count(): > # stop after midnight > now = datetime.utcnow() > last_midnight = now.replace(hour=0, minute=0, > second=0, microsecond=0) > seconds_since_midnight = (now - last_midnight).total_seconds() > if 5 <= seconds_since_midnight <= 15: > break > session.execute(simple_insert, [i, i]) > result = session.execute("SELECT bar, WRITETIME(bar) FROM test;") > {code} > EDIT: This behavior occurs with server-generated timestamps; in this > particular test, I set {{use_client_timestamp}} to {{False}}. > Under normal circumstances, the values and writetimes would increase > together, but when inserted over the leap second, they don't. These {{value, > writetime}} pairs are sorted by writetime: > {code} > (582, 1435708799285000) > (579, 1435708799339000) > (583, 1435708799593000) > (580, 1435708799643000) > (584, 1435708799897000) > (581, 1435708799958000) > {code} > The values were inserted in increasing order, but their writetimes are in a > different order because of the repeated second. During the first instance of > 23:59:59, the values 579, 580, and 581 were inserted at the beginning, > middle, and end of the second. During the leap second, which is also > 23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and > end of the second. However, since the two seconds are the same second, they > appear interleaved with respect to timestamps, as shown above. > So, should I consider this behavior correct? If not, how should Cassandra > correctly handle the discontinuity introduced by the insertion of a leap > second? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[ https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500443#comment-14500443 ] Andy Tolbert commented on CASSANDRA-9131: - [~benedict], thanks. I agree if we could have a client timestamp implementation in the drivers that is monotonically increasing while avoiding the possibility of having operations with the same timestamp would be ideal. I agree with [~slebresne] that a reference algorithm would be very helpful so all drivers implement this in a specific way. I'll bring up this topic to the team. I've seen it brought up in a number of issues with differing opinions, is the current perspective that going forward client-provided timestamps will be preferred over server timestamps? Depending on the perspective I can see it being more important for clients to properly implement this. The current behavior in all the drivers to my knowledge is that using client timestamps has to be explicit and only the python and java drivers have a way to enable it for all queries. > Defining correct behavior during leap second insertion > -- > > Key: CASSANDRA-9131 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9131 > Project: Cassandra > Issue Type: Bug > Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue > Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux >Reporter: Jim Witschey >Assignee: Jim Witschey > > On Linux platforms, the insertion of a leap second breaks the monotonicity of > timestamps. This can make values appear to have been inserted into Cassandra > in a different order than they were. I want to know what behavior is expected > and desirable for inserts over this discontinuity. > From a timestamp perspective, an inserted leap second looks like a repeat of > the previous second: > {code} > $ while true ; do echo "`date +%s%N` `date -u`" ; sleep .5 ; done > 1435708798171327029 Tue Jun 30 23:59:58 UTC 2015 > 1435708798679392477 Tue Jun 30 23:59:58 UTC 2015 > 1435708799187550335 Tue Jun 30 23:59:59 UTC 2015 > 1435708799695670453 Tue Jun 30 23:59:59 UTC 2015 > 1435708799203902068 Tue Jun 30 23:59:59 UTC 2015 > 1435708799712168566 Tue Jun 30 23:59:59 UTC 2015 > 1435708800220473932 Wed Jul 1 00:00:00 UTC 2015 > 1435708800728908190 Wed Jul 1 00:00:00 UTC 2015 > 1435708801237611983 Wed Jul 1 00:00:01 UTC 2015 > 1435708801746251996 Wed Jul 1 00:00:01 UTC 2015 > {code} > Note that 23:59:59 repeats itself, and that the timestamps increase during > the first time through, then step back down to the beginning of the second > and increase again. > As a result, the timestamps on values inserted during these seconds will be > out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and > synced them to shortly before the leap second would be inserted. During the > insertion of the leap second, I ran a test with logic something like: > {code} > simple_insert = session.prepare( > 'INSERT INTO test (foo, bar) VALUES (?, ?);') > for i in itertools.count(): > # stop after midnight > now = datetime.utcnow() > last_midnight = now.replace(hour=0, minute=0, > second=0, microsecond=0) > seconds_since_midnight = (now - last_midnight).total_seconds() > if 5 <= seconds_since_midnight <= 15: > break > session.execute(simple_insert, [i, i]) > result = session.execute("SELECT bar, WRITETIME(bar) FROM test;") > {code} > EDIT: This behavior occurs with server-generated timestamps; in this > particular test, I set {{use_client_timestamp}} to {{False}}. > Under normal circumstances, the values and writetimes would increase > together, but when inserted over the leap second, they don't. These {{value, > writetime}} pairs are sorted by writetime: > {code} > (582, 1435708799285000) > (579, 1435708799339000) > (583, 1435708799593000) > (580, 1435708799643000) > (584, 1435708799897000) > (581, 1435708799958000) > {code} > The values were inserted in increasing order, but their writetimes are in a > different order because of the repeated second. During the first instance of > 23:59:59, the values 579, 580, and 581 were inserted at the beginning, > middle, and end of the second. During the leap second, which is also > 23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and > end of the second. However, since the two seconds are the same second, they > appear interleaved with respect to timestamps, as shown above. > So, should I consider this behavior correct? If not, how should Cassandra > correctly handle the discontinuity introduced by the insertion of a leap > second? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[ https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500259#comment-14500259 ] Sylvain Lebresne commented on CASSANDRA-9131: - bq. if we should offer some sample code Yes, I suspect it would be appreciated if we were to provide some kind of example/reference algorithm for this (maybe just in form of some pseudo-code), probably in the protocol spec documentation. > Defining correct behavior during leap second insertion > -- > > Key: CASSANDRA-9131 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9131 > Project: Cassandra > Issue Type: Bug > Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue > Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux >Reporter: Jim Witschey >Assignee: Jim Witschey > > On Linux platforms, the insertion of a leap second breaks the monotonicity of > timestamps. This can make values appear to have been inserted into Cassandra > in a different order than they were. I want to know what behavior is expected > and desirable for inserts over this discontinuity. > From a timestamp perspective, an inserted leap second looks like a repeat of > the previous second: > {code} > $ while true ; do echo "`date +%s%N` `date -u`" ; sleep .5 ; done > 1435708798171327029 Tue Jun 30 23:59:58 UTC 2015 > 1435708798679392477 Tue Jun 30 23:59:58 UTC 2015 > 1435708799187550335 Tue Jun 30 23:59:59 UTC 2015 > 1435708799695670453 Tue Jun 30 23:59:59 UTC 2015 > 1435708799203902068 Tue Jun 30 23:59:59 UTC 2015 > 1435708799712168566 Tue Jun 30 23:59:59 UTC 2015 > 1435708800220473932 Wed Jul 1 00:00:00 UTC 2015 > 1435708800728908190 Wed Jul 1 00:00:00 UTC 2015 > 1435708801237611983 Wed Jul 1 00:00:01 UTC 2015 > 1435708801746251996 Wed Jul 1 00:00:01 UTC 2015 > {code} > Note that 23:59:59 repeats itself, and that the timestamps increase during > the first time through, then step back down to the beginning of the second > and increase again. > As a result, the timestamps on values inserted during these seconds will be > out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and > synced them to shortly before the leap second would be inserted. During the > insertion of the leap second, I ran a test with logic something like: > {code} > simple_insert = session.prepare( > 'INSERT INTO test (foo, bar) VALUES (?, ?);') > for i in itertools.count(): > # stop after midnight > now = datetime.utcnow() > last_midnight = now.replace(hour=0, minute=0, > second=0, microsecond=0) > seconds_since_midnight = (now - last_midnight).total_seconds() > if 5 <= seconds_since_midnight <= 15: > break > session.execute(simple_insert, [i, i]) > result = session.execute("SELECT bar, WRITETIME(bar) FROM test;") > {code} > EDIT: This behavior occurs with server-generated timestamps; in this > particular test, I set {{use_client_timestamp}} to {{False}}. > Under normal circumstances, the values and writetimes would increase > together, but when inserted over the leap second, they don't. These {{value, > writetime}} pairs are sorted by writetime: > {code} > (582, 1435708799285000) > (579, 1435708799339000) > (583, 1435708799593000) > (580, 1435708799643000) > (584, 1435708799897000) > (581, 1435708799958000) > {code} > The values were inserted in increasing order, but their writetimes are in a > different order because of the repeated second. During the first instance of > 23:59:59, the values 579, 580, and 581 were inserted at the beginning, > middle, and end of the second. During the leap second, which is also > 23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and > end of the second. However, since the two seconds are the same second, they > appear interleaved with respect to timestamps, as shown above. > So, should I consider this behavior correct? If not, how should Cassandra > correctly handle the discontinuity introduced by the insertion of a leap > second? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[ https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500186#comment-14500186 ] Benedict commented on CASSANDRA-9131: - [~andrew.tolbert]: if you feel like taking a look at CASSANDRA-6106, this could be a useful approach for the java-driver (and be adapted to other drivers). Whether or not it uses the microsecond time is kind of irrelevant to the point at hand (although potentially also helpful in itself), but the approach to staggering the time corrections is very applicable. This would prevent the "only have fewer than 1000 inserts in a leap second" problem, because the 1s shift backwards in time would be spread over the proceeding minute, with each second taking around 20ms longer to elapse than they otherwise would. Either my or Sylvain's method of updating the clock time would suffice, and be a tremendous improvement to behaviour here in the drivers. > Defining correct behavior during leap second insertion > -- > > Key: CASSANDRA-9131 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9131 > Project: Cassandra > Issue Type: Bug > Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue > Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux >Reporter: Jim Witschey >Assignee: Jim Witschey > > On Linux platforms, the insertion of a leap second breaks the monotonicity of > timestamps. This can make values appear to have been inserted into Cassandra > in a different order than they were. I want to know what behavior is expected > and desirable for inserts over this discontinuity. > From a timestamp perspective, an inserted leap second looks like a repeat of > the previous second: > {code} > $ while true ; do echo "`date +%s%N` `date -u`" ; sleep .5 ; done > 1435708798171327029 Tue Jun 30 23:59:58 UTC 2015 > 1435708798679392477 Tue Jun 30 23:59:58 UTC 2015 > 1435708799187550335 Tue Jun 30 23:59:59 UTC 2015 > 1435708799695670453 Tue Jun 30 23:59:59 UTC 2015 > 1435708799203902068 Tue Jun 30 23:59:59 UTC 2015 > 1435708799712168566 Tue Jun 30 23:59:59 UTC 2015 > 1435708800220473932 Wed Jul 1 00:00:00 UTC 2015 > 1435708800728908190 Wed Jul 1 00:00:00 UTC 2015 > 1435708801237611983 Wed Jul 1 00:00:01 UTC 2015 > 1435708801746251996 Wed Jul 1 00:00:01 UTC 2015 > {code} > Note that 23:59:59 repeats itself, and that the timestamps increase during > the first time through, then step back down to the beginning of the second > and increase again. > As a result, the timestamps on values inserted during these seconds will be > out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and > synced them to shortly before the leap second would be inserted. During the > insertion of the leap second, I ran a test with logic something like: > {code} > simple_insert = session.prepare( > 'INSERT INTO test (foo, bar) VALUES (?, ?);') > for i in itertools.count(): > # stop after midnight > now = datetime.utcnow() > last_midnight = now.replace(hour=0, minute=0, > second=0, microsecond=0) > seconds_since_midnight = (now - last_midnight).total_seconds() > if 5 <= seconds_since_midnight <= 15: > break > session.execute(simple_insert, [i, i]) > result = session.execute("SELECT bar, WRITETIME(bar) FROM test;") > {code} > EDIT: This behavior occurs with server-generated timestamps; in this > particular test, I set {{use_client_timestamp}} to {{False}}. > Under normal circumstances, the values and writetimes would increase > together, but when inserted over the leap second, they don't. These {{value, > writetime}} pairs are sorted by writetime: > {code} > (582, 1435708799285000) > (579, 1435708799339000) > (583, 1435708799593000) > (580, 1435708799643000) > (584, 1435708799897000) > (581, 1435708799958000) > {code} > The values were inserted in increasing order, but their writetimes are in a > different order because of the repeated second. During the first instance of > 23:59:59, the values 579, 580, and 581 were inserted at the beginning, > middle, and end of the second. During the leap second, which is also > 23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and > end of the second. However, since the two seconds are the same second, they > appear interleaved with respect to timestamps, as shown above. > So, should I consider this behavior correct? If not, how should Cassandra > correctly handle the discontinuity introduced by the insertion of a leap > second? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[ https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500127#comment-14500127 ] Andy Tolbert commented on CASSANDRA-9131: - {quote} if we should update the client protocol spec/docs to make clear that this problem exists, and that clients are expected to work around it if we should offer some sample code, and work with the Java Driver team to ensure this problem doesn't affect it {quote} This is something I'm actively looking at on the drivers side for each driver. As of now only the python-driver and java-driver have a mechanism to enable automatically set client timestamps. The other drivers that support 2.1 have a mechanism of the client specifying the timestamp (they could also use 'USING TIMESTAMP' i suppose), so it will be up to user / time implementation. Most of the drivers will have an active means of setting client timestamp through API by June 30th 2015. * python-driver: [Session use_client_timestamp|http://datastax.github.io/python-driver/api/cassandra/cluster.html?highlight=timestamp#cassandra.cluster.Session.use_client_timestamp] not monotonic, uses time.time(). * java-driver: [TimestampGenerator|http://www.datastax.com/drivers/java/2.1/com/datastax/driver/core/TimestampGenerator.html]. All client implementations are monotonic, but client can provide their own. If > 999 entries for same millisecond, will reloop over the same millisecond, so its not completely monotonic. Possible problem if > 1000 entries during leap second ([code|https://github.com/datastax/java-driver/blob/2.1/driver-core/src/main/java/com/datastax/driver/core/AbstractTimestampGenerator.java]) > Defining correct behavior during leap second insertion > -- > > Key: CASSANDRA-9131 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9131 > Project: Cassandra > Issue Type: Bug > Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue > Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux >Reporter: Jim Witschey >Assignee: Jim Witschey > > On Linux platforms, the insertion of a leap second breaks the monotonicity of > timestamps. This can make values appear to have been inserted into Cassandra > in a different order than they were. I want to know what behavior is expected > and desirable for inserts over this discontinuity. > From a timestamp perspective, an inserted leap second looks like a repeat of > the previous second: > {code} > $ while true ; do echo "`date +%s%N` `date -u`" ; sleep .5 ; done > 1435708798171327029 Tue Jun 30 23:59:58 UTC 2015 > 1435708798679392477 Tue Jun 30 23:59:58 UTC 2015 > 1435708799187550335 Tue Jun 30 23:59:59 UTC 2015 > 1435708799695670453 Tue Jun 30 23:59:59 UTC 2015 > 1435708799203902068 Tue Jun 30 23:59:59 UTC 2015 > 1435708799712168566 Tue Jun 30 23:59:59 UTC 2015 > 1435708800220473932 Wed Jul 1 00:00:00 UTC 2015 > 1435708800728908190 Wed Jul 1 00:00:00 UTC 2015 > 1435708801237611983 Wed Jul 1 00:00:01 UTC 2015 > 1435708801746251996 Wed Jul 1 00:00:01 UTC 2015 > {code} > Note that 23:59:59 repeats itself, and that the timestamps increase during > the first time through, then step back down to the beginning of the second > and increase again. > As a result, the timestamps on values inserted during these seconds will be > out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and > synced them to shortly before the leap second would be inserted. During the > insertion of the leap second, I ran a test with logic something like: > {code} > simple_insert = session.prepare( > 'INSERT INTO test (foo, bar) VALUES (?, ?);') > for i in itertools.count(): > # stop after midnight > now = datetime.utcnow() > last_midnight = now.replace(hour=0, minute=0, > second=0, microsecond=0) > seconds_since_midnight = (now - last_midnight).total_seconds() > if 5 <= seconds_since_midnight <= 15: > break > session.execute(simple_insert, [i, i]) > result = session.execute("SELECT bar, WRITETIME(bar) FROM test;") > {code} > EDIT: This behavior occurs with server-generated timestamps; in this > particular test, I set {{use_client_timestamp}} to {{False}}. > Under normal circumstances, the values and writetimes would increase > together, but when inserted over the leap second, they don't. These {{value, > writetime}} pairs are sorted by writetime: > {code} > (582, 1435708799285000) > (579, 1435708799339000) > (583, 1435708799593000) > (580, 1435708799643000) > (584, 1435708799897000) > (581, 1435708799958000) > {code} > The values were inserted in increasing order, but their writetimes are in a > different order because of the repeated second. During the first instance of > 23:59:59, the values 579, 580, and 581
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[ https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500058#comment-14500058 ] Robert Stupp commented on CASSANDRA-9131: - bq. problem with server-side timestamps is unsafe retry by clients in the event of a failure. got it > Defining correct behavior during leap second insertion > -- > > Key: CASSANDRA-9131 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9131 > Project: Cassandra > Issue Type: Bug > Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue > Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux >Reporter: Jim Witschey >Assignee: Jim Witschey > > On Linux platforms, the insertion of a leap second breaks the monotonicity of > timestamps. This can make values appear to have been inserted into Cassandra > in a different order than they were. I want to know what behavior is expected > and desirable for inserts over this discontinuity. > From a timestamp perspective, an inserted leap second looks like a repeat of > the previous second: > {code} > $ while true ; do echo "`date +%s%N` `date -u`" ; sleep .5 ; done > 1435708798171327029 Tue Jun 30 23:59:58 UTC 2015 > 1435708798679392477 Tue Jun 30 23:59:58 UTC 2015 > 1435708799187550335 Tue Jun 30 23:59:59 UTC 2015 > 1435708799695670453 Tue Jun 30 23:59:59 UTC 2015 > 1435708799203902068 Tue Jun 30 23:59:59 UTC 2015 > 1435708799712168566 Tue Jun 30 23:59:59 UTC 2015 > 1435708800220473932 Wed Jul 1 00:00:00 UTC 2015 > 1435708800728908190 Wed Jul 1 00:00:00 UTC 2015 > 1435708801237611983 Wed Jul 1 00:00:01 UTC 2015 > 1435708801746251996 Wed Jul 1 00:00:01 UTC 2015 > {code} > Note that 23:59:59 repeats itself, and that the timestamps increase during > the first time through, then step back down to the beginning of the second > and increase again. > As a result, the timestamps on values inserted during these seconds will be > out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and > synced them to shortly before the leap second would be inserted. During the > insertion of the leap second, I ran a test with logic something like: > {code} > simple_insert = session.prepare( > 'INSERT INTO test (foo, bar) VALUES (?, ?);') > for i in itertools.count(): > # stop after midnight > now = datetime.utcnow() > last_midnight = now.replace(hour=0, minute=0, > second=0, microsecond=0) > seconds_since_midnight = (now - last_midnight).total_seconds() > if 5 <= seconds_since_midnight <= 15: > break > session.execute(simple_insert, [i, i]) > result = session.execute("SELECT bar, WRITETIME(bar) FROM test;") > {code} > EDIT: This behavior occurs with server-generated timestamps; in this > particular test, I set {{use_client_timestamp}} to {{False}}. > Under normal circumstances, the values and writetimes would increase > together, but when inserted over the leap second, they don't. These {{value, > writetime}} pairs are sorted by writetime: > {code} > (582, 1435708799285000) > (579, 1435708799339000) > (583, 1435708799593000) > (580, 1435708799643000) > (584, 1435708799897000) > (581, 1435708799958000) > {code} > The values were inserted in increasing order, but their writetimes are in a > different order because of the repeated second. During the first instance of > 23:59:59, the values 579, 580, and 581 were inserted at the beginning, > middle, and end of the second. During the leap second, which is also > 23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and > end of the second. However, since the two seconds are the same second, they > appear interleaved with respect to timestamps, as shown above. > So, should I consider this behavior correct? If not, how should Cassandra > correctly handle the discontinuity introduced by the insertion of a leap > second? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[ https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500011#comment-14500011 ] Benedict commented on CASSANDRA-9131: - The problem with server-side timestamps is unsafe retry by clients in the event of a failure. CASSANDRA-6106 is a reference point, in that this provided both a wrapper around microsecond resolution as well as a staggered application of shifts in system clock time (also ensuring it never went backwards). > Defining correct behavior during leap second insertion > -- > > Key: CASSANDRA-9131 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9131 > Project: Cassandra > Issue Type: Bug > Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue > Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux >Reporter: Jim Witschey >Assignee: Jim Witschey > > On Linux platforms, the insertion of a leap second breaks the monotonicity of > timestamps. This can make values appear to have been inserted into Cassandra > in a different order than they were. I want to know what behavior is expected > and desirable for inserts over this discontinuity. > From a timestamp perspective, an inserted leap second looks like a repeat of > the previous second: > {code} > $ while true ; do echo "`date +%s%N` `date -u`" ; sleep .5 ; done > 1435708798171327029 Tue Jun 30 23:59:58 UTC 2015 > 1435708798679392477 Tue Jun 30 23:59:58 UTC 2015 > 1435708799187550335 Tue Jun 30 23:59:59 UTC 2015 > 1435708799695670453 Tue Jun 30 23:59:59 UTC 2015 > 1435708799203902068 Tue Jun 30 23:59:59 UTC 2015 > 1435708799712168566 Tue Jun 30 23:59:59 UTC 2015 > 1435708800220473932 Wed Jul 1 00:00:00 UTC 2015 > 1435708800728908190 Wed Jul 1 00:00:00 UTC 2015 > 1435708801237611983 Wed Jul 1 00:00:01 UTC 2015 > 1435708801746251996 Wed Jul 1 00:00:01 UTC 2015 > {code} > Note that 23:59:59 repeats itself, and that the timestamps increase during > the first time through, then step back down to the beginning of the second > and increase again. > As a result, the timestamps on values inserted during these seconds will be > out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and > synced them to shortly before the leap second would be inserted. During the > insertion of the leap second, I ran a test with logic something like: > {code} > simple_insert = session.prepare( > 'INSERT INTO test (foo, bar) VALUES (?, ?);') > for i in itertools.count(): > # stop after midnight > now = datetime.utcnow() > last_midnight = now.replace(hour=0, minute=0, > second=0, microsecond=0) > seconds_since_midnight = (now - last_midnight).total_seconds() > if 5 <= seconds_since_midnight <= 15: > break > session.execute(simple_insert, [i, i]) > result = session.execute("SELECT bar, WRITETIME(bar) FROM test;") > {code} > EDIT: This behavior occurs with server-generated timestamps; in this > particular test, I set {{use_client_timestamp}} to {{False}}. > Under normal circumstances, the values and writetimes would increase > together, but when inserted over the leap second, they don't. These {{value, > writetime}} pairs are sorted by writetime: > {code} > (582, 1435708799285000) > (579, 1435708799339000) > (583, 1435708799593000) > (580, 1435708799643000) > (584, 1435708799897000) > (581, 1435708799958000) > {code} > The values were inserted in increasing order, but their writetimes are in a > different order because of the repeated second. During the first instance of > 23:59:59, the values 579, 580, and 581 were inserted at the beginning, > middle, and end of the second. During the leap second, which is also > 23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and > end of the second. However, since the two seconds are the same second, they > appear interleaved with respect to timestamps, as shown above. > So, should I consider this behavior correct? If not, how should Cassandra > correctly handle the discontinuity introduced by the insertion of a leap > second? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[ https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1453#comment-1453 ] Robert Stupp commented on CASSANDRA-9131: - Hm. I'm not convinced of client side timestamps. IMO maintaining timestamps on the clients (either these are ”fat clients” or other servers) just moves the problem to an area that might not be sensitive for correct system wall clock (e.g. NTP). I've seen operations handling Win and Linux environments completely separate - but both worlds with a constant time drift of several minutes (not funny). I'm not completely against client provided timestamps - but would prefer to make that an optional feature (i.e. move {{TIMESTAMP xx}} to the protocol). TL;DR just want to throw in an idea: We could encapsulate {{System.currentTimeMillis()}} - if we detect that the clock went backwards, we slow down ”our system clock” and vice versa if the clock moves forwards. > Defining correct behavior during leap second insertion > -- > > Key: CASSANDRA-9131 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9131 > Project: Cassandra > Issue Type: Bug > Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue > Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux >Reporter: Jim Witschey >Assignee: Jim Witschey > > On Linux platforms, the insertion of a leap second breaks the monotonicity of > timestamps. This can make values appear to have been inserted into Cassandra > in a different order than they were. I want to know what behavior is expected > and desirable for inserts over this discontinuity. > From a timestamp perspective, an inserted leap second looks like a repeat of > the previous second: > {code} > $ while true ; do echo "`date +%s%N` `date -u`" ; sleep .5 ; done > 1435708798171327029 Tue Jun 30 23:59:58 UTC 2015 > 1435708798679392477 Tue Jun 30 23:59:58 UTC 2015 > 1435708799187550335 Tue Jun 30 23:59:59 UTC 2015 > 1435708799695670453 Tue Jun 30 23:59:59 UTC 2015 > 1435708799203902068 Tue Jun 30 23:59:59 UTC 2015 > 1435708799712168566 Tue Jun 30 23:59:59 UTC 2015 > 1435708800220473932 Wed Jul 1 00:00:00 UTC 2015 > 1435708800728908190 Wed Jul 1 00:00:00 UTC 2015 > 1435708801237611983 Wed Jul 1 00:00:01 UTC 2015 > 1435708801746251996 Wed Jul 1 00:00:01 UTC 2015 > {code} > Note that 23:59:59 repeats itself, and that the timestamps increase during > the first time through, then step back down to the beginning of the second > and increase again. > As a result, the timestamps on values inserted during these seconds will be > out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and > synced them to shortly before the leap second would be inserted. During the > insertion of the leap second, I ran a test with logic something like: > {code} > simple_insert = session.prepare( > 'INSERT INTO test (foo, bar) VALUES (?, ?);') > for i in itertools.count(): > # stop after midnight > now = datetime.utcnow() > last_midnight = now.replace(hour=0, minute=0, > second=0, microsecond=0) > seconds_since_midnight = (now - last_midnight).total_seconds() > if 5 <= seconds_since_midnight <= 15: > break > session.execute(simple_insert, [i, i]) > result = session.execute("SELECT bar, WRITETIME(bar) FROM test;") > {code} > EDIT: This behavior occurs with server-generated timestamps; in this > particular test, I set {{use_client_timestamp}} to {{False}}. > Under normal circumstances, the values and writetimes would increase > together, but when inserted over the leap second, they don't. These {{value, > writetime}} pairs are sorted by writetime: > {code} > (582, 1435708799285000) > (579, 1435708799339000) > (583, 1435708799593000) > (580, 1435708799643000) > (584, 1435708799897000) > (581, 1435708799958000) > {code} > The values were inserted in increasing order, but their writetimes are in a > different order because of the repeated second. During the first instance of > 23:59:59, the values 579, 580, and 581 were inserted at the beginning, > middle, and end of the second. During the leap second, which is also > 23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and > end of the second. However, since the two seconds are the same second, they > appear interleaved with respect to timestamps, as shown above. > So, should I consider this behavior correct? If not, how should Cassandra > correctly handle the discontinuity introduced by the insertion of a leap > second? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[ https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14499963#comment-14499963 ] Benedict commented on CASSANDRA-9131: - I can't recall entirely, but I don't think that was quite the way the conversation resolved. It's not exactly "expected" behaviour on either side, but obviously there's nothing we can do about clients that are subjected to this bug, and we intend to deprecate server-side timestamps (if perhaps never eliminate them entirely). So the question is # if we should fix server side timestamps by making the clock universally monotonically increasing (as opposed to only per-client connection) which would at least somewhat mitigate this problem (but not eliminate it entirely, for any sequence of inserts hitting multiple servers) # if we should update the client protocol spec/docs to make clear that this problem exists, and that clients are expected to work around it ## if we should offer some sample code, and work with the Java Driver team to ensure this problem doesn't affect it > Defining correct behavior during leap second insertion > -- > > Key: CASSANDRA-9131 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9131 > Project: Cassandra > Issue Type: Bug > Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue > Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux >Reporter: Jim Witschey >Assignee: Jim Witschey > > On Linux platforms, the insertion of a leap second breaks the monotonicity of > timestamps. This can make values appear to have been inserted into Cassandra > in a different order than they were. I want to know what behavior is expected > and desirable for inserts over this discontinuity. > From a timestamp perspective, an inserted leap second looks like a repeat of > the previous second: > {code} > $ while true ; do echo "`date +%s%N` `date -u`" ; sleep .5 ; done > 1435708798171327029 Tue Jun 30 23:59:58 UTC 2015 > 1435708798679392477 Tue Jun 30 23:59:58 UTC 2015 > 1435708799187550335 Tue Jun 30 23:59:59 UTC 2015 > 1435708799695670453 Tue Jun 30 23:59:59 UTC 2015 > 1435708799203902068 Tue Jun 30 23:59:59 UTC 2015 > 1435708799712168566 Tue Jun 30 23:59:59 UTC 2015 > 1435708800220473932 Wed Jul 1 00:00:00 UTC 2015 > 1435708800728908190 Wed Jul 1 00:00:00 UTC 2015 > 1435708801237611983 Wed Jul 1 00:00:01 UTC 2015 > 1435708801746251996 Wed Jul 1 00:00:01 UTC 2015 > {code} > Note that 23:59:59 repeats itself, and that the timestamps increase during > the first time through, then step back down to the beginning of the second > and increase again. > As a result, the timestamps on values inserted during these seconds will be > out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and > synced them to shortly before the leap second would be inserted. During the > insertion of the leap second, I ran a test with logic something like: > {code} > simple_insert = session.prepare( > 'INSERT INTO test (foo, bar) VALUES (?, ?);') > for i in itertools.count(): > # stop after midnight > now = datetime.utcnow() > last_midnight = now.replace(hour=0, minute=0, > second=0, microsecond=0) > seconds_since_midnight = (now - last_midnight).total_seconds() > if 5 <= seconds_since_midnight <= 15: > break > session.execute(simple_insert, [i, i]) > result = session.execute("SELECT bar, WRITETIME(bar) FROM test;") > {code} > EDIT: This behavior occurs with server-generated timestamps; in this > particular test, I set {{use_client_timestamp}} to {{False}}. > Under normal circumstances, the values and writetimes would increase > together, but when inserted over the leap second, they don't. These {{value, > writetime}} pairs are sorted by writetime: > {code} > (582, 1435708799285000) > (579, 1435708799339000) > (583, 1435708799593000) > (580, 1435708799643000) > (584, 1435708799897000) > (581, 1435708799958000) > {code} > The values were inserted in increasing order, but their writetimes are in a > different order because of the repeated second. During the first instance of > 23:59:59, the values 579, 580, and 581 were inserted at the beginning, > middle, and end of the second. During the leap second, which is also > 23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and > end of the second. However, since the two seconds are the same second, they > appear interleaved with respect to timestamps, as shown above. > So, should I consider this behavior correct? If not, how should Cassandra > correctly handle the discontinuity introduced by the insertion of a leap > second? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[ https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485842#comment-14485842 ] Jim Witschey commented on CASSANDRA-9131: - To sum up some conversation on #cassandra-dev: This is expected behavior for server-generated timestamps. Client-generated timestamps will help, but won't solve the problem completely. That's expected behavior as well. Is that right? If so, feel free to close. > Defining correct behavior during leap second insertion > -- > > Key: CASSANDRA-9131 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9131 > Project: Cassandra > Issue Type: Bug > Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue > Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux >Reporter: Jim Witschey > > On Linux platforms, the insertion of a leap second breaks the monotonicity of > timestamps. This can make values appear to have been inserted into Cassandra > in a different order than they were. I want to know what behavior is expected > and desirable for inserts over this discontinuity. > From a timestamp perspective, an inserted leap second looks like a repeat of > the previous second: > {code} > $ while true ; do echo "`date +%s%N` `date -u`" ; sleep .5 ; done > 1435708798171327029 Tue Jun 30 23:59:58 UTC 2015 > 1435708798679392477 Tue Jun 30 23:59:58 UTC 2015 > 1435708799187550335 Tue Jun 30 23:59:59 UTC 2015 > 1435708799695670453 Tue Jun 30 23:59:59 UTC 2015 > 1435708799203902068 Tue Jun 30 23:59:59 UTC 2015 > 1435708799712168566 Tue Jun 30 23:59:59 UTC 2015 > 1435708800220473932 Wed Jul 1 00:00:00 UTC 2015 > 1435708800728908190 Wed Jul 1 00:00:00 UTC 2015 > 1435708801237611983 Wed Jul 1 00:00:01 UTC 2015 > 1435708801746251996 Wed Jul 1 00:00:01 UTC 2015 > {code} > Note that 23:59:59 repeats itself, and that the timestamps increase during > the first time through, then step back down to the beginning of the second > and increase again. > As a result, the timestamps on values inserted during these seconds will be > out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and > synced them to shortly before the leap second would be inserted. During the > insertion of the leap second, I ran a test with logic something like: > {code} > simple_insert = session.prepare( > 'INSERT INTO test (foo, bar) VALUES (?, ?);') > for i in itertools.count(): > # stop after midnight > now = datetime.utcnow() > last_midnight = now.replace(hour=0, minute=0, > second=0, microsecond=0) > seconds_since_midnight = (now - last_midnight).total_seconds() > if 5 <= seconds_since_midnight <= 15: > break > session.execute(simple_insert, [i, i]) > result = session.execute("SELECT bar, WRITETIME(bar) FROM test;") > {code} > EDIT: This behavior occurs with server-generated timestamps; in this > particular test, I set {{use_client_timestamp}} to {{False}}. > Under normal circumstances, the values and writetimes would increase > together, but when inserted over the leap second, they don't. These {{value, > writetime}} pairs are sorted by writetime: > {code} > (582, 1435708799285000) > (579, 1435708799339000) > (583, 1435708799593000) > (580, 1435708799643000) > (584, 1435708799897000) > (581, 1435708799958000) > {code} > The values were inserted in increasing order, but their writetimes are in a > different order because of the repeated second. During the first instance of > 23:59:59, the values 579, 580, and 581 were inserted at the beginning, > middle, and end of the second. During the leap second, which is also > 23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and > end of the second. However, since the two seconds are the same second, they > appear interleaved with respect to timestamps, as shown above. > So, should I consider this behavior correct? If not, how should Cassandra > correctly handle the discontinuity introduced by the insertion of a leap > second? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[ https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485398#comment-14485398 ] T Jake Luciani commented on CASSANDRA-9131: --- System.currentTimeMillis() uses CLOCK_REALTIME and will include leap seconds. > Defining correct behavior during leap second insertion > -- > > Key: CASSANDRA-9131 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9131 > Project: Cassandra > Issue Type: Bug > Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue > Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux >Reporter: Jim Witschey > > On Linux platforms, the insertion of a leap second breaks the monotonicity of > timestamps. This can make values appear to have been inserted into Cassandra > in a different order than they were. I want to know what behavior is expected > and desirable for inserts over this discontinuity. > From a timestamp perspective, an inserted leap second looks like a repeat of > the previous second: > {code} > $ while true ; do echo "`date +%s%N` `date -u`" ; sleep .5 ; done > 1435708798171327029 Tue Jun 30 23:59:58 UTC 2015 > 1435708798679392477 Tue Jun 30 23:59:58 UTC 2015 > 1435708799187550335 Tue Jun 30 23:59:59 UTC 2015 > 1435708799695670453 Tue Jun 30 23:59:59 UTC 2015 > 1435708799203902068 Tue Jun 30 23:59:59 UTC 2015 > 1435708799712168566 Tue Jun 30 23:59:59 UTC 2015 > 1435708800220473932 Wed Jul 1 00:00:00 UTC 2015 > 1435708800728908190 Wed Jul 1 00:00:00 UTC 2015 > 1435708801237611983 Wed Jul 1 00:00:01 UTC 2015 > 1435708801746251996 Wed Jul 1 00:00:01 UTC 2015 > {code} > Note that 23:59:59 repeats itself, and that the timestamps increase during > the first time through, then step back down to the beginning of the second > and increase again. > As a result, the timestamps on values inserted during these seconds will be > out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and > synced them to shortly before the leap second would be inserted. During the > insertion of the leap second, I ran a test with logic something like: > {code} > simple_insert = session.prepare( > 'INSERT INTO test (foo, bar) VALUES (?, ?);') > for i in itertools.count(): > # stop after midnight > now = datetime.utcnow() > last_midnight = now.replace(hour=0, minute=0, > second=0, microsecond=0) > seconds_since_midnight = (now - last_midnight).total_seconds() > if 5 <= seconds_since_midnight <= 15: > break > session.execute(simple_insert, [i, i]) > result = session.execute("SELECT bar, WRITETIME(bar) FROM test;") > {code} > EDIT: This behavior occurs with server-generated timestamps; in this > particular test, I set {{use_client_timestamp}} to {{False}}. > Under normal circumstances, the values and writetimes would increase > together, but when inserted over the leap second, they don't. These {{value, > writetime}} pairs are sorted by writetime: > {code} > (582, 1435708799285000) > (579, 1435708799339000) > (583, 1435708799593000) > (580, 1435708799643000) > (584, 1435708799897000) > (581, 1435708799958000) > {code} > The values were inserted in increasing order, but their writetimes are in a > different order because of the repeated second. During the first instance of > 23:59:59, the values 579, 580, and 581 were inserted at the beginning, > middle, and end of the second. During the leap second, which is also > 23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and > end of the second. However, since the two seconds are the same second, they > appear interleaved with respect to timestamps, as shown above. > So, should I consider this behavior correct? If not, how should Cassandra > correctly handle the discontinuity introduced by the insertion of a leap > second? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[ https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485331#comment-14485331 ] Jim Witschey commented on CASSANDRA-9131: - Client-side timestamps don't have anything to do with this particular test; the behavior described happens with {{use_client_timestamp = False}}. I'll revise the ticket to specify that. > Defining correct behavior during leap second insertion > -- > > Key: CASSANDRA-9131 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9131 > Project: Cassandra > Issue Type: Bug > Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue > Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux >Reporter: Jim Witschey > > On Linux platforms, the insertion of a leap second breaks the monotonicity of > timestamps. This can make values appear to have been inserted into Cassandra > in a different order than they were. I want to know what behavior is expected > and desirable for inserts over this discontinuity. > From a timestamp perspective, an inserted leap second looks like a repeat of > the previous second: > {code} > $ while true ; do echo "`date +%s%N` `date -u`" ; sleep .5 ; done > 1435708798171327029 Tue Jun 30 23:59:58 UTC 2015 > 1435708798679392477 Tue Jun 30 23:59:58 UTC 2015 > 1435708799187550335 Tue Jun 30 23:59:59 UTC 2015 > 1435708799695670453 Tue Jun 30 23:59:59 UTC 2015 > 1435708799203902068 Tue Jun 30 23:59:59 UTC 2015 > 1435708799712168566 Tue Jun 30 23:59:59 UTC 2015 > 1435708800220473932 Wed Jul 1 00:00:00 UTC 2015 > 1435708800728908190 Wed Jul 1 00:00:00 UTC 2015 > 1435708801237611983 Wed Jul 1 00:00:01 UTC 2015 > 1435708801746251996 Wed Jul 1 00:00:01 UTC 2015 > {code} > Note that 23:59:59 repeats itself, and that the timestamps increase during > the first time through, then step back down to the beginning of the second > and increase again. > As a result, the timestamps on values inserted during these seconds will be > out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and > synced them to shortly before the leap second would be inserted. During the > insertion of the leap second, I ran a test with logic something like: > {code} > simple_insert = session.prepare( > 'INSERT INTO test (foo, bar) VALUES (?, ?);') > for i in itertools.count(): > # stop after midnight > now = datetime.utcnow() > last_midnight = now.replace(hour=0, minute=0, > second=0, microsecond=0) > seconds_since_midnight = (now - last_midnight).total_seconds() > if 5 <= seconds_since_midnight <= 15: > break > session.execute(simple_insert, [i, i]) > result = session.execute("SELECT bar, WRITETIME(bar) FROM test;") > {code} > Under normal circumstances, the values and writetimes would increase > together, but when inserted over the leap second, they don't. These {{value, > writetime}} pairs are sorted by writetime: > {code} > (582, 1435708799285000) > (579, 1435708799339000) > (583, 1435708799593000) > (580, 1435708799643000) > (584, 1435708799897000) > (581, 1435708799958000) > {code} > The values were inserted in increasing order, but their writetimes are in a > different order because of the repeated second. During the first instance of > 23:59:59, the values 579, 580, and 581 were inserted at the beginning, > middle, and end of the second. During the leap second, which is also > 23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and > end of the second. However, since the two seconds are the same second, they > appear interleaved with respect to timestamps, as shown above. > So, should I consider this behavior correct? If not, how should Cassandra > correctly handle the discontinuity introduced by the insertion of a leap > second? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[ https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485308#comment-14485308 ] Jonathan Ellis commented on CASSANDRA-9131: --- This is a good example of why I'm not sure pushing timestamps to the client and saying "here, you figure it out" is such a great idea. /cc [~slebresne] > Defining correct behavior during leap second insertion > -- > > Key: CASSANDRA-9131 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9131 > Project: Cassandra > Issue Type: Bug > Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue > Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux >Reporter: Jim Witschey > > On Linux platforms, the insertion of a leap second breaks the monotonicity of > timestamps. This can make values appear to have been inserted into Cassandra > in a different order than they were. I want to know what behavior is expected > and desirable for inserts over this discontinuity. > From a timestamp perspective, an inserted leap second looks like a repeat of > the previous second: > {code} > $ while true ; do echo "`date +%s%N` `date -u`" ; sleep .5 ; done > 1435708798171327029 Tue Jun 30 23:59:58 UTC 2015 > 1435708798679392477 Tue Jun 30 23:59:58 UTC 2015 > 1435708799187550335 Tue Jun 30 23:59:59 UTC 2015 > 1435708799695670453 Tue Jun 30 23:59:59 UTC 2015 > 1435708799203902068 Tue Jun 30 23:59:59 UTC 2015 > 1435708799712168566 Tue Jun 30 23:59:59 UTC 2015 > 1435708800220473932 Wed Jul 1 00:00:00 UTC 2015 > 1435708800728908190 Wed Jul 1 00:00:00 UTC 2015 > 1435708801237611983 Wed Jul 1 00:00:01 UTC 2015 > 1435708801746251996 Wed Jul 1 00:00:01 UTC 2015 > {code} > Note that 23:59:59 repeats itself, and that the timestamps increase during > the first time through, then step back down to the beginning of the second > and increase again. > As a result, the timestamps on values inserted during these seconds will be > out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and > synced them to shortly before the leap second would be inserted. During the > insertion of the leap second, I ran a test with logic something like: > {code} > simple_insert = session.prepare( > 'INSERT INTO test (foo, bar) VALUES (?, ?);') > for i in itertools.count(): > # stop after midnight > now = datetime.utcnow() > last_midnight = now.replace(hour=0, minute=0, > second=0, microsecond=0) > seconds_since_midnight = (now - last_midnight).total_seconds() > if 5 <= seconds_since_midnight <= 15: > break > session.execute(simple_insert, [i, i]) > result = session.execute("SELECT bar, WRITETIME(bar) FROM test;") > {code} > Under normal circumstances, the values and writetimes would increase > together, but when inserted over the leap second, they don't. These {{value, > writetime}} pairs are sorted by writetime: > {code} > (582, 1435708799285000) > (579, 1435708799339000) > (583, 1435708799593000) > (580, 1435708799643000) > (584, 1435708799897000) > (581, 1435708799958000) > {code} > The values were inserted in increasing order, but their writetimes are in a > different order because of the repeated second. During the first instance of > 23:59:59, the values 579, 580, and 581 were inserted at the beginning, > middle, and end of the second. During the leap second, which is also > 23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and > end of the second. However, since the two seconds are the same second, they > appear interleaved with respect to timestamps, as shown above. > So, should I consider this behavior correct? If not, how should Cassandra > correctly handle the discontinuity introduced by the insertion of a leap > second? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[ https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14484991#comment-14484991 ] Benedict commented on CASSANDRA-9131: - Huh. I guess I misunderstood Unix time until now. The problem we're going to have here is that server-side timestamps are being deprecated, and this problem is being pushed onto clients so that they can also perform safe retry of non-idempotent writes (like counters). CASSANDRA-6106 was deprecated and would have resolved this particular problem by ensuring the clock used by Cassandra was monotonically increasing. I'm not sure I would want a solution of just making the clock monotonically increasing, as we would get some potentially problematic bunching during steps backwards/forwards (6106 attempts to spread any movement in time across a period of a few seconds). I'm not sure we have any other good solution to this, on either the client or server side.. > Defining correct behavior during leap second insertion > -- > > Key: CASSANDRA-9131 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9131 > Project: Cassandra > Issue Type: Bug > Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue > Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux >Reporter: Jim Witschey > > On Linux platforms, the insertion of a leap second breaks the monotonicity of > timestamps. This can make values appear to have been inserted into Cassandra > in a different order than they were. I want to know what behavior is expected > and desirable for inserts over this discontinuity. > From a timestamp perspective, an inserted leap second looks like a repeat of > the previous second: > {code} > $ while true ; do echo "`date +%s%N` `date -u`" ; sleep .5 ; done > 1435708798171327029 Tue Jun 30 23:59:58 UTC 2015 > 1435708798679392477 Tue Jun 30 23:59:58 UTC 2015 > 1435708799187550335 Tue Jun 30 23:59:59 UTC 2015 > 1435708799695670453 Tue Jun 30 23:59:59 UTC 2015 > 1435708799203902068 Tue Jun 30 23:59:59 UTC 2015 > 1435708799712168566 Tue Jun 30 23:59:59 UTC 2015 > 1435708800220473932 Wed Jul 1 00:00:00 UTC 2015 > 1435708800728908190 Wed Jul 1 00:00:00 UTC 2015 > 1435708801237611983 Wed Jul 1 00:00:01 UTC 2015 > 1435708801746251996 Wed Jul 1 00:00:01 UTC 2015 > {code} > Note that 23:59:59 repeats itself, and that the timestamps increase during > the first time through, then step back down to the beginning of the second > and increase again. > As a result, the timestamps on values inserted during these seconds will be > out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and > synced them to shortly before the leap second would be inserted. During the > insertion of the leap second, I ran a test with logic something like: > {code} > simple_insert = session.prepare( > 'INSERT INTO test (foo, bar) VALUES (?, ?);') > for i in itertools.count(): > # stop after midnight > now = datetime.utcnow() > last_midnight = now.replace(hour=0, minute=0, > second=0, microsecond=0) > seconds_since_midnight = (now - last_midnight).total_seconds() > if 5 <= seconds_since_midnight <= 15: > break > session.execute(simple_insert, [i, i]) > result = session.execute("SELECT bar, WRITETIME(bar) FROM test;") > {code} > Under normal circumstances, the values and writetimes would increase > together, but when inserted over the leap second, they don't. These {{value, > writetime}} pairs are sorted by writetime: > {code} > (582, 1435708799285000) > (579, 1435708799339000) > (583, 1435708799593000) > (580, 1435708799643000) > (584, 1435708799897000) > (581, 1435708799958000) > {code} > The values were inserted in increasing order, but their writetimes are in a > different order because of the repeated second. During the first instance of > 23:59:59, the values 579, 580, and 581 were inserted at the beginning, > middle, and end of the second. During the leap second, which is also > 23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and > end of the second. However, since the two seconds are the same second, they > appear interleaved with respect to timestamps, as shown above. > So, should I consider this behavior correct? If not, how should Cassandra > correctly handle the discontinuity introduced by the insertion of a leap > second? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[ https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14484583#comment-14484583 ] Philip Thompson commented on CASSANDRA-9131: I think this discussion is worth bringing up at #cassandra-dev as well. > Defining correct behavior during leap second insertion > -- > > Key: CASSANDRA-9131 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9131 > Project: Cassandra > Issue Type: Bug > Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue > Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux >Reporter: Jim Witschey > > On Linux platforms, the insertion of a leap second breaks the monotonicity of > timestamps. This can make values appear to have been inserted into Cassandra > in a different order than they were. I want to know what behavior is expected > and desirable for inserts over this discontinuity. > From a timestamp perspective, an inserted leap second looks like a repeat of > the previous second: > {code} > $ while true ; do echo "`date +%s%N` `date -u`" ; sleep .5 ; done > 1435708798171327029 Tue Jun 30 23:59:58 UTC 2015 > 1435708798679392477 Tue Jun 30 23:59:58 UTC 2015 > 1435708799187550335 Tue Jun 30 23:59:59 UTC 2015 > 1435708799695670453 Tue Jun 30 23:59:59 UTC 2015 > 1435708799203902068 Tue Jun 30 23:59:59 UTC 2015 > 1435708799712168566 Tue Jun 30 23:59:59 UTC 2015 > 1435708800220473932 Wed Jul 1 00:00:00 UTC 2015 > 1435708800728908190 Wed Jul 1 00:00:00 UTC 2015 > 1435708801237611983 Wed Jul 1 00:00:01 UTC 2015 > 1435708801746251996 Wed Jul 1 00:00:01 UTC 2015 > {code} > Note that 23:59:59 repeats itself, and that the timestamps increase during > the first time through, then step back down to the beginning of the second > and increase again. > As a result, the timestamps on values inserted during these seconds will be > out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and > synced them to shortly before the leap second would be inserted. During the > insertion of the leap second, I ran a test with logic something like: > {code} > simple_insert = session.prepare( > 'INSERT INTO test (foo, bar) VALUES (?, ?);') > for i in itertools.count(): > # stop after midnight > now = datetime.utcnow() > last_midnight = now.replace(hour=0, minute=0, > second=0, microsecond=0) > seconds_since_midnight = (now - last_midnight).total_seconds() > if 5 <= seconds_since_midnight <= 15: > break > session.execute(simple_insert, [i, i]) > result = session.execute("SELECT bar, WRITETIME(bar) FROM test;") > {code} > Under normal circumstances, the values and writetimes would increase > together, but when inserted over the leap second, they don't. These {{value, > writetime}} pairs are sorted by writetime: > {code} > (582, 1435708799285000) > (579, 1435708799339000) > (583, 1435708799593000) > (580, 1435708799643000) > (584, 1435708799897000) > (581, 1435708799958000) > {code} > The values were inserted in increasing order, but their writetimes are in a > different order because of the repeated second. During the first instance of > 23:59:59, the values 579, 580, and 581 were inserted at the beginning, > middle, and end of the second. During the leap second, which is also > 23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and > end of the second. However, since the two seconds are the same second, they > appear interleaved with respect to timestamps, as shown above. > So, should I consider this behavior correct? If not, how should Cassandra > correctly handle the discontinuity introduced by the insertion of a leap > second? -- This message was sent by Atlassian JIRA (v6.3.4#6332)