[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835822#comment-17835822 ] Brandon Williams edited comment on CASSANDRA-19448 at 4/10/24 10:43 PM: One problem is that CommitLogArchiverTest seems to still be flaky on trunk. This also seems to have made flaky the DropRecreateAndRestoreTest.testCreateWithIdRestore unit test on all branches. was (Author: brandon.williams): One problem is that CommitLogArchiverTest seems to still be flaky on trunk. > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Assignee: Maxwell Guo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835802#comment-17835802 ] Brandon Williams edited comment on CASSANDRA-19448 at 4/10/24 4:08 PM: --- Looks good to me, let's check CI: ||Branch||CI|| |[4.0|https://github.com/driftx/cassandra/tree/CASSANDRA-19448-4.0]|[j8|https://app.circleci.com/pipelines/github/driftx/cassandra/1579/workflows/03c0b2f6-d84f-440d-baad-bf323810a292], [j11|https://app.circleci.com/pipelines/github/driftx/cassandra/1579/workflows/ec2044d0-157a-4f21-a9a2-8a95214cf28f]| |[4.1|https://github.com/driftx/cassandra/tree/CASSANDRA-19448-4.1]|[j8|https://app.circleci.com/pipelines/github/driftx/cassandra/1582/workflows/aa8c4b45-683e-4c55-80b3-a6923fa6338c], [j11|https://app.circleci.com/pipelines/github/driftx/cassandra/1582/workflows/0eb028d6-65d4-4a80-bd39-04d2eff73d10]| |[5.0|https://github.com/driftx/cassandra/tree/CASSANDRA-19448-5.0]|[j11|https://app.circleci.com/pipelines/github/driftx/cassandra/1578/workflows/9caf1884-b1c5-44af-8413-87f1aa164396], [j17|https://app.circleci.com/pipelines/github/driftx/cassandra/1578/workflows/5d2777fc-b33e-48a3-8e4f-739df29f3656]| |[trunk|https://github.com/driftx/cassandra/tree/CASSANDRA-19448-trunk]|[j11|https://app.circleci.com/pipelines/github/driftx/cassandra/1580/workflows/5c6cfe0b-c421-4112-a886-a3fbfda51d54], [j17|https://app.circleci.com/pipelines/github/driftx/cassandra/1580/workflows/c177c474-33d6-42b4-b88a-a72fc6abe8b5]| was (Author: brandon.williams): Looks good to me, let's check CI: ||Branch||CI|| |[4.0|https://github.com/driftx/cassandra/tree/CASSANDRA-19448-4.0]|[j8|https://app.circleci.com/pipelines/github/driftx/cassandra/1579/workflows/03c0b2f6-d84f-440d-baad-bf323810a292], [j11|https://app.circleci.com/pipelines/github/driftx/cassandra/1579/workflows/ec2044d0-157a-4f21-a9a2-8a95214cf28f]| |[4.1|https://github.com/driftx/cassandra/tree/CASSANDRA-19448-4.1]|[j8|https://app.circleci.com/pipelines/github/driftx/cassandra/1577/workflows/4a17a2f1-c20d-4f5a-84f3-d26cec580fe5], [j11|https://app.circleci.com/pipelines/github/driftx/cassandra/1577/workflows/1e9d317a-73cc-4b24-82ba-31e351aac796]| |[5.0|https://github.com/driftx/cassandra/tree/CASSANDRA-19448-5.0]|[j11|https://app.circleci.com/pipelines/github/driftx/cassandra/1578/workflows/9caf1884-b1c5-44af-8413-87f1aa164396], [j17|https://app.circleci.com/pipelines/github/driftx/cassandra/1578/workflows/5d2777fc-b33e-48a3-8e4f-739df29f3656]| |[trunk|https://github.com/driftx/cassandra/tree/CASSANDRA-19448-trunk]|[j11|https://app.circleci.com/pipelines/github/driftx/cassandra/1580/workflows/5c6cfe0b-c421-4112-a886-a3fbfda51d54], [j17|https://app.circleci.com/pipelines/github/driftx/cassandra/1580/workflows/c177c474-33d6-42b4-b88a-a72fc6abe8b5]| > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Assignee: Maxwell Guo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands,
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835570#comment-17835570 ] Maxwell Guo edited comment on CASSANDRA-19448 at 4/10/24 6:28 AM: -- I asked [~cnlwsu], he agreed with your suggestion. [~brandon.williams] bq. I suggest we leave precision alone and deprecate it in 4.0 and 4.1 and remove in 5.0 and trunk. I updated the patch , and deprecated precision in 4.0 and 4.1 for the api in commitlogMbean, and removed it in 5.0 and trunk. was (Author: maxwellguo): I asked [~cnlwsu], he agreed with your suggestion. [~brandon.williams] > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Assignee: Maxwell Guo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835570#comment-17835570 ] Maxwell Guo edited comment on CASSANDRA-19448 at 4/10/24 2:14 AM: -- I asked [~cnlwsu], he agreed with your suggestion. [~brandon.williams] was (Author: maxwellguo): I asked [~cnlwsu], he agreed with your suggestion. > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Assignee: Maxwell Guo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835570#comment-17835570 ] Maxwell Guo edited comment on CASSANDRA-19448 at 4/10/24 2:13 AM: -- I asked [~cnlwsu], he agreed with your suggestion. was (Author: maxwellguo): I asked [~cnlwsu], he agreed with your suggestion. > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Assignee: Maxwell Guo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835523#comment-17835523 ] Brandon Williams edited comment on CASSANDRA-19448 at 4/9/24 8:20 PM: -- Let's see. Perhaps we can gain a second committer to review as well. was (Author: brandon.williams): Let's see. Perhaps we can gain a second reviewer as well. > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Assignee: Maxwell Guo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835466#comment-17835466 ] Brandon Williams edited comment on CASSANDRA-19448 at 4/9/24 3:44 PM: -- I'm sorry that I missed that removing the precision would affect JMX, and I can see you fixed the [JMXCompatibility test|https://github.com/apache/cassandra/pull/3238/commits/b0db7522d34d82a67a8d1ef486394c0aee86f7f2]. Unfortunately we can't do it like this, CASSANDRA-8734 exposed these archiver methods, very likely for some automation. We should deprecate this first, however in this instance we aren't removing functionality so much as obviating the need for precision, so any breakage should be minimal. I suggest we leave precision alone and deprecate it in 4.0 and 4.1 and remove in 5.0 and trunk. WDYT? was (Author: brandon.williams): I'm sorry that I missed that removing the precision would affect JMX, and I can see you fixed the [JMXCompatibility test|https://github.com/apache/cassandra/pull/3238/commits/b0db7522d34d82a67a8d1ef486394c0aee86f7f2]. Unfortunately we can't do it like this, CASSANDRA-8734 exposed these archiver methods, very likely for some automation. We should deprecate this first, however in this instance we aren't removing functionality so much as obviating the need for precision, so any breakage should be minimal. I suggest we leave precision alone in 4.0, deprecate in 4.1 and remove in 5.0 and trunk. WDYT? > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Assignee: Maxwell Guo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835164#comment-17835164 ] Maxwell Guo edited comment on CASSANDRA-19448 at 4/9/24 2:44 PM: - [~brandon.williams] I fixed the failure and remove the precision that you have mentioned . But the CI may be slow because of the limited resource .:( ||Heading 1||Heading 2|| |[4.0|https://github.com/apache/cassandra/pull/3238]| [java8|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/572/workflows/949a3a12-ac70-40ab-b935-185d2a6c51e3] [java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/572/workflows/12ebdd83-1a87-4b2c-9806-261ce54c1bb1]| |[4.1|https://github.com/apache/cassandra/pull/3237]| [java8|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/573/workflows/8983cd7b-8a35-4b06-81b4-bef66aa02ab7] [java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/573/workflows/eeffd27e-227c-4720-865f-e0bc7c563496]| |[5.0|https://github.com/apache/cassandra/pull/3236]| [java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/574/workflows/08679195-4afd-4ff2-95e3-ea7800e2145a] [java17|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/574/workflows/c889a4f2-76d9-4dc5-97b2-ce51f6c60405] | |[trunk|https://github.com/apache/cassandra/pull/3215]| [java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/575/workflows/276a267b-315d-41d2-a216-3ba2912fdb9f] [java17|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/575/workflows/9fab5111-8209-4889-b839-006d02c4f3ce] | was (Author: maxwellguo): [~brandon.williams] I fixed the failure and remove the precision that you have mentioned . But the CI may be slow because of the limited resource .:( ||Heading 1||Heading 2|| |[4.0|https://github.com/apache/cassandra/pull/3238]| [java8|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/571/workflows/bb88be9e-4745-4ed0-b7d1-a101cd76c913] [java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/571/workflows/dfb90672-e5a8-44bf-9d14-e3d5e5bb3934]| |[4.1|https://github.com/apache/cassandra/pull/3237]| [java8|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/570/workflows/c9d44874-9ddd-4618-b464-136c8c94e3f7] [java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/570/workflows/dc8e1c13-84c8-4032-9a03-34f7220a7379]| |[5.0|https://github.com/apache/cassandra/pull/3236]| [java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/569/workflows/2221edd7-60bc-4c5d-bccc-a6bea1abc7ba] [java17|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/569/workflows/6b94b590-bdbf-4250-a7f7-f6b78c38292d] | |[trunk|https://github.com/apache/cassandra/pull/3215]| [java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/568/workflows/12d9bbbd-fb19-477d-92e0-5534e0da652e] [java17|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/568/workflows/5aad317f-cf3b-411a-8c68-8910cb52d005] | > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Assignee: Maxwell Guo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835164#comment-17835164 ] Maxwell Guo edited comment on CASSANDRA-19448 at 4/9/24 6:27 AM: - [~brandon.williams] I fixed the failure and remove the precision that you have mentioned . But the CI may be slow because of the limited resource .:( ||Heading 1||Heading 2|| |[4.0|https://github.com/apache/cassandra/pull/3238]| [java8|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/571/workflows/bb88be9e-4745-4ed0-b7d1-a101cd76c913] [java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/571/workflows/dfb90672-e5a8-44bf-9d14-e3d5e5bb3934]| |[4.1|https://github.com/apache/cassandra/pull/3237]| [java8|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/570/workflows/c9d44874-9ddd-4618-b464-136c8c94e3f7] [java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/570/workflows/dc8e1c13-84c8-4032-9a03-34f7220a7379]| |[5.0|https://github.com/apache/cassandra/pull/3236]| [java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/569/workflows/2221edd7-60bc-4c5d-bccc-a6bea1abc7ba] [java17|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/569/workflows/6b94b590-bdbf-4250-a7f7-f6b78c38292d] | |[trunk|https://github.com/apache/cassandra/pull/3215]| [java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/568/workflows/12d9bbbd-fb19-477d-92e0-5534e0da652e] [java17|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/568/workflows/5aad317f-cf3b-411a-8c68-8910cb52d005] | was (Author: maxwellguo): [~brandon.williams] I fixed the failure and remove the precision that you have mentioned 。 ||Heading 1||Heading 2|| |[4.0|https://github.com/apache/cassandra/pull/3238]| [java8|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/571/workflows/bb88be9e-4745-4ed0-b7d1-a101cd76c913] [java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/571/workflows/dfb90672-e5a8-44bf-9d14-e3d5e5bb3934]| |[4.1|https://github.com/apache/cassandra/pull/3237]| [java8|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/570/workflows/c9d44874-9ddd-4618-b464-136c8c94e3f7] [java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/570/workflows/dc8e1c13-84c8-4032-9a03-34f7220a7379]| |[5.0|https://github.com/apache/cassandra/pull/3236]| [java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/569/workflows/2221edd7-60bc-4c5d-bccc-a6bea1abc7ba] [java17|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/569/workflows/6b94b590-bdbf-4250-a7f7-f6b78c38292d] | |[trunk|https://github.com/apache/cassandra/pull/3215]| [java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/568/workflows/12d9bbbd-fb19-477d-92e0-5534e0da652e] [java17|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/568/workflows/5aad317f-cf3b-411a-8c68-8910cb52d005] | > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Assignee: Maxwell Guo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17834997#comment-17834997 ] Brandon Williams edited comment on CASSANDRA-19448 at 4/8/24 7:15 PM: -- I left one small note about a lingering todo that I can remove on commit if you agree. ||Branch||CI|| |[4.0|https://github.com/driftx/cassandra/tree/CASSANDRA-19448-4.0]|[j8|https://app.circleci.com/pipelines/github/driftx/cassandra/1566/workflows/4fae134a-fb02-4b6e-bb97-dd781651b9ef], [j11|https://app.circleci.com/pipelines/github/driftx/cassandra/1566/workflows/49ac5ce2-0a63-48b5-bc41-da30073dfd73]| |[4.1|https://github.com/driftx/cassandra/tree/CASSANDRA-19448-4.1]|[j8|https://app.circleci.com/pipelines/github/driftx/cassandra/1567/workflows/efaf1236-111d-44cd-bee7-f0cffcc99986], [j11|https://app.circleci.com/pipelines/github/driftx/cassandra/1567/workflows/e9c2a9b3-2516-4b11-b701-5077144afe75]| |[5.0|https://github.com/driftx/cassandra/tree/CASSANDRA-19448-5.0]|[j11|https://app.circleci.com/pipelines/github/driftx/cassandra/1569/workflows/601b0002-7536-4343-8499-e30368d4432d], [j17|https://app.circleci.com/pipelines/github/driftx/cassandra/1569/workflows/839908dc-5622-46da-bd04-4152a1305a43]| |[trunk|https://github.com/driftx/cassandra/tree/CASSANDRA-19448-trunk]|[j11|https://app.circleci.com/pipelines/github/driftx/cassandra/1570/workflows/247ebd77-4fe6-4d08-b95d-6c6e75ef99fa], [j17|https://app.circleci.com/pipelines/github/driftx/cassandra/1570/workflows/628377f0-98d7-4531-baa3-429ca5319f01]| was (Author: brandon.williams): I left one small note about a lingering todo that I can remove on commit if you agree. ||Branch||CI|| |[4.0|https://github.com/driftx/cassandra/tree/CASSANDRA-19448-4.0]|[j8|https://app.circleci.com/pipelines/github/driftx/cassandra/1566/workflows/4fae134a-fb02-4b6e-bb97-dd781651b9ef], [j11|https://app.circleci.com/pipelines/github/driftx/cassandra/1566/workflows/49ac5ce2-0a63-48b5-bc41-da30073dfd73]| |[4.1|https://github.com/driftx/cassandra/tree/CASSANDRA-19448-4.1]|[j8|https://app.circleci.com/pipelines/github/driftx/cassandra/1567/workflows/efaf1236-111d-44cd-bee7-f0cffcc99986], [j11|https://app.circleci.com/pipelines/github/driftx/cassandra/1567/workflows/e9c2a9b3-2516-4b11-b701-5077144afe75]| |[5.0|https://github.com/driftx/cassandra/tree/CASSANDRA-19448-5.0]|[j11|https://app.circleci.com/pipelines/github/driftx/cassandra/1569/workflows/601b0002-7536-4343-8499-e30368d4432d], [j17|https://app.circleci.com/pipelines/github/driftx/cassandra/1569/workflows/839908dc-5622-46da-bd04-4152a1305a43]| |[trunk|https://github.com/driftx/cassandra/tree/CASSANDRA-19448-trunk]|[j11|https://app.circleci.com/pipelines/github/driftx/cassandra/1568/workflows/52d7f8aa-b2ee-425f-ad8e-a031ed6f7526], [j17|https://app.circleci.com/pipelines/github/driftx/cassandra/1568/workflows/ed32dc18-3c16-4661-aaf8-570fcd48d608]| > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Assignee: Maxwell Guo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17834975#comment-17834975 ] Maxwell Guo edited comment on CASSANDRA-19448 at 4/8/24 3:57 PM: - ||Heading 1||Heading 2|| |trunk |[trunk|https://github.com/apache/cassandra/pull/3215]| |5.0|[5.0|https://github.com/apache/cassandra/pull/3236]| |4.1|[4.1|https://github.com/apache/cassandra/pull/3237]| |4.0|[4.0|https://github.com/apache/cassandra/pull/3238]| cc [~brandon.williams] was (Author: maxwellguo): ||Heading 1||Heading 2|| |trunk |[trunk|https://github.com/apache/cassandra/pull/3215]| |5.0|[5.0|https://github.com/apache/cassandra/pull/3236]| |4.1|[4.1|https://github.com/apache/cassandra/pull/3237]| ||4.0|[4.0|https://github.com/apache/cassandra/pull/3238]| > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Assignee: Maxwell Guo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17834892#comment-17834892 ] Maxwell Guo edited comment on CASSANDRA-19448 at 4/8/24 12:31 PM: -- Thanks [~brandon.williams]， I will update the prs for other branches latter today.:) was (Author: maxwellguo): Thanks [~brandon.williams] > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Assignee: Maxwell Guo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832369#comment-17832369 ] Maxwell Guo edited comment on CASSANDRA-19448 at 4/3/24 6:12 AM: - Hi Sorry for the late reply. I was in a hurry recently so I was away for a while. [~brandon.williams][~tiagomlalves] [~jeromatron], I have also made a PR for this ,And I might want a more general solution to this problem, whether rip is seconds, milliseconds, or microseconds. I have just prepare the pr but have not been tested very complete, here is the pr , and I will update the CI after they finished. Besides, I used to pay more attention to whether this category is a bug or an improvement, because it is related to whether to prepare a copy for 4.x. ||Heading 1||Heading 2|| |PR for trunk |[pr|https://github.com/apache/cassandra/pull/3215/]| And the ci are running ,will check if the they finished. ||Heading 1||Heading 2|| |java11|[java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/553/workflows/a4f3412e-833d-4d50-878d-851bd6bdd94a]| |java17|[java17|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/553/workflows/f61b7b2f-e239-4620-af85-af68186a6227]| Update: fix build error for unused imports was (Author: maxwellguo): Hi Sorry for the late reply. I was in a hurry recently so I was away for a while. [~brandon.williams][~tiagomlalves] [~jeromatron], I have also made a PR for this ,And I might want a more general solution to this problem, whether rip is seconds, milliseconds, or microseconds. I have just prepare the pr but have not been tested very complete, here is the pr , and I will update the CI after they finished. Besides, I used to pay more attention to whether this category is a bug or an improvement, because it is related to whether to prepare a copy for 4.x. ||Heading 1||Heading 2|| |PR for trunk |[pr|https://github.com/apache/cassandra/pull/3215/]| And the ci are running ,will check if the they finished. ||Heading 1||Heading 2|| |java11|[java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/552/workflows/48484065-581e-4a9b-9187-7ad8429306eb]| |java17|[java17|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/552/workflows/0a5999af-fc19-4743-886b-1ef749c49395]| Update: fix build error for unused imports > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Assignee: Maxwell Guo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832369#comment-17832369 ] Maxwell Guo edited comment on CASSANDRA-19448 at 4/3/24 3:06 AM: - Hi Sorry for the late reply. I was in a hurry recently so I was away for a while. [~brandon.williams][~tiagomlalves] [~jeromatron], I have also made a PR for this ,And I might want a more general solution to this problem, whether rip is seconds, milliseconds, or microseconds. I have just prepare the pr but have not been tested very complete, here is the pr , and I will update the CI after they finished. Besides, I used to pay more attention to whether this category is a bug or an improvement, because it is related to whether to prepare a copy for 4.x. ||Heading 1||Heading 2|| |PR for trunk |[pr|https://github.com/apache/cassandra/pull/3215/]| And the ci are running ,will check if the they finished. ||Heading 1||Heading 2|| |java11|[java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/552/workflows/48484065-581e-4a9b-9187-7ad8429306eb]| |java17|[java17|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/552/workflows/0a5999af-fc19-4743-886b-1ef749c49395]| Update: fix build error for unused imports was (Author: maxwellguo): Hi Sorry for the late reply. I was in a hurry recently so I was away for a while. [~brandon.williams][~tiagomlalves] [~jeromatron], I have also made a PR for this ,And I might want a more general solution to this problem, whether rip is seconds, milliseconds, or microseconds. I have just prepare the pr but have not been tested very complete, here is the pr , and I will update the CI after they finished. Besides, I used to pay more attention to whether this category is a bug or an improvement, because it is related to whether to prepare a copy for 4.x. ||Heading 1||Heading 2|| |PR for trunk |[pr|https://github.com/apache/cassandra/pull/3215/]| And the ci are running ,will check if the they finished. ||Heading 1||Heading 2|| |java11|[java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/549/workflows/f1110dc1-b08e-4db5-97bf-e945658dc28b]| |java17|[java17|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/549/workflows/e9cf230a-9b78-4568-9ae4-14c0e68510a4]| Update: fix build error for unused imports > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Assignee: Maxwell Guo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833365#comment-17833365 ] Maxwell Guo edited comment on CASSANDRA-19448 at 4/3/24 3:00 AM: - [~tiagomlalves]Thanks for you reply : bq. What I miss is the practical knowledge on how a customer can determine at microsecond level precision if something is being inserted in a cluster. In my mind, as for this , seconds and millseconds may meet the same question , and microseconds can more accurately express whether data is needed . 1、RIP is an abbreviation for error， and I have fixed it, thanks for reminding; 2、As for RIPLEVEL enum class, I classified it intentionally. If you think this code is redundant, I can change it. bq. I would be willing to to pick up where you left in https://github.com/apache/cassandra/pull/3215 and try out that simplification if you agree. of course you can , but before that you can see my my latest update. https://github.com/apache/cassandra/pull/3215/ was (Author: maxwellguo): [~tiagomlalves]Thanks for you reply : bq. What I miss is the practical knowledge on how a customer can determine at microsecond level precision if something is being inserted in a cluster. In my mind, as for this , seconds and millseconds may meet the same question , and microseconds can more accurately express whether data is needed . 1、RIP is an abbreviation for error， and I have fixed it, thanks for reminding; 2、As for RIPLEVEL enum class, I classified it intentionally. If you think this code is redundant, I can change it. bq. I would be willing to to pick up where you left in https://github.com/apache/cassandra/pull/3215 and try out that simplification if you agree. of course you can , but before that you can see my my latest update. https://github.com/apache/cassandra/pull/3215/ > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Assignee: Maxwell Guo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[
https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832992#comment-17832992
]
Maxwell Guo edited comment on CASSANDRA-19448 at 4/2/24 6:25 AM:
-
[~brandon.williams][~tiagomlalves]Thank you very much for your reply.
Regarding the granularity of RIP, my main point is that it mainly depends on
the needs of the user. My patch allows the user to select seconds, milliseconds
and microseconds. All I need to do is to ensure that it is consistent with
Cassandra 's timestamp itself.
bq. What I wonder is, in which scenarios would microsecond-level PIT restore
would be useful?
[~brandon.williams] may have already described it clearly, but what I may want
to say again is, if a batch of data is deleted by mistake, then accurate time
granularity to microseconds is the only prerequisite to ensure that all data
can be restored in c*. Milliseconds and seconds are not enough, they may lost
data.
bq.couldn't we detect automatically the granularity of PIT restore based on the
value
I will update the PR again, and [~brandon.williams][~tiagomlalves] If you are
willing, can you be the reviewer :)
And I will prepare PRs for other branch if the fix for trunk is accepted .
Update:
bq.Regarding the code changes, couldn't we detect automatically the granularity
of PIT restore based on the value an user specifies? We could make our parsing
more lenient by just doing `DateTimeFormatter.ofPattern(":MM:dd
HH:mm:ss[.[SS][SSS]]")` allowing to parse all seconds, milliseconds, and
microseconds.
See [description for rip
|https://github.com/Maxwell-Guo/cassandra/blob/CASSANDRA-19448/conf/commitlog_archiving.properties#L43]
, the user can randomly configures the rip time. I do not restrict the user to
manually specify the accuracy level for Seconds, millseconds or microseconds .
That's to say the code detect automatically the granularity of PIT restore
based on the value an user specifies.
As for
bq. Also, if we could leverage on `Instant` datatype instead of `long` in
`CommitLogArchiver` and postpone conversions to `CommitLogRestorer`
I made some optimizations to the code based on your suggestions, but I find
that changing to Instant is not fundamentally different from the original long
type, because
[restorePointInTime|https://github.com/Maxwell-Guo/cassandra/blob/CASSANDRA-19448/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L123]
is only used for long value comparison, and as we have said the the
restorePointInTime should be in microsecond level, so I change it to micro
level
[here|https://github.com/Maxwell-Guo/cassandra/blob/CASSANDRA-19448/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L207]
was (Author: maxwellguo):
[~brandon.williams][~tiagomlalves]Thank you very much for your reply.
Regarding the granularity of RIP, my main point is that it mainly depends on
the needs of the user. My patch allows the user to select seconds, milliseconds
and microseconds. All I need to do is to ensure that it is consistent with
Cassandra 's timestamp itself.
bq. What I wonder is, in which scenarios would microsecond-level PIT restore
would be useful?
[~brandon.williams] may have already described it clearly, but what I may want
to say again is, if a batch of data is deleted by mistake, then accurate time
granularity to microseconds is the only prerequisite to ensure that all data
can be restored in c*. Milliseconds and seconds are not enough, they may lost
data.
bq.couldn't we detect automatically the granularity of PIT restore based on the
value
I will update the PR again, and [~brandon.williams][~tiagomlalves] If you are
willing, can you be the reviewer :)
And I will prepare PRs for other branch if the fix for trunk is accepted .
> CommitlogArchiver only has granularity to seconds for restore_point_in_time
> ---
>
> Key: CASSANDRA-19448
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19448
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Commit Log
>Reporter: Jeremy Hanna
>Assignee: Maxwell Guo
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Commitlog archiver allows users to backup commitlog files for the purpose of
> doing point in time restores. The [configuration
> file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties]
> gives an example of down to the seconds granularity but then asks what
> whether the timestamps are microseconds or milliseconds - defaulting to
> microseconds. Because the [CommitLogArchiver uses a second based date
>
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832992#comment-17832992 ] Maxwell Guo edited comment on CASSANDRA-19448 at 4/2/24 2:23 AM: - [~brandon.williams][~tiagomlalves]Thank you very much for your reply. Regarding the granularity of RIP, my main point is that it mainly depends on the needs of the user. My patch allows the user to select seconds, milliseconds and microseconds. All I need to do is to ensure that it is consistent with Cassandra 's timestamp itself. bq. What I wonder is, in which scenarios would microsecond-level PIT restore would be useful? [~brandon.williams] may have already described it clearly, but what I may want to say again is, if a batch of data is deleted by mistake, then accurate time granularity to microseconds is the only prerequisite to ensure that all data can be restored in c*. Milliseconds and seconds are not enough, they may lost data. bq.couldn't we detect automatically the granularity of PIT restore based on the value I will update the PR again, and [~brandon.williams][~tiagomlalves] If you are willing, can you be the reviewer :) And I will prepare PRs for other branch if the fix for trunk is accepted . was (Author: maxwellguo): [~brandon.williams][~tiagomlalves]Thank you very much for your reply. Regarding the granularity of RIP, my main point is that it mainly depends on the needs of the user. My patch allows the user to select seconds, milliseconds and microseconds. All I need to do is to ensure that it is consistent with Cassandra 's timestamp itself. bq. What I wonder is, in which scenarios would microsecond-level PIT restore would be useful? [~brandon.williams] may have already described it clearly, but what I may want to say again is, if a batch of data is deleted by mistake, then accurate time granularity to microseconds is the only prerequisite to ensure that all data can be restored in c*. Milliseconds and seconds are not enough, they may lost data. bq.couldn't we detect automatically the granularity of PIT restore based on the value I will update the PR again, and [~brandon.williams][~tiagomlalves] If you are willing, can you be the reviewer :) I will prepare PRs for other branch if the fix for trunk is accepted . > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Assignee: Maxwell Guo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832992#comment-17832992 ] Maxwell Guo edited comment on CASSANDRA-19448 at 4/2/24 2:23 AM: - [~brandon.williams][~tiagomlalves]Thank you very much for your reply. Regarding the granularity of RIP, my main point is that it mainly depends on the needs of the user. My patch allows the user to select seconds, milliseconds and microseconds. All I need to do is to ensure that it is consistent with Cassandra 's timestamp itself. bq. What I wonder is, in which scenarios would microsecond-level PIT restore would be useful? [~brandon.williams] may have already described it clearly, but what I may want to say again is, if a batch of data is deleted by mistake, then accurate time granularity to microseconds is the only prerequisite to ensure that all data can be restored in c*. Milliseconds and seconds are not enough, they may lost data. bq.couldn't we detect automatically the granularity of PIT restore based on the value I will update the PR again, and [~brandon.williams][~tiagomlalves] If you are willing, can you be the reviewer :) I will prepare PRs for other branch if the fix for trunk is accepted . was (Author: maxwellguo): [~brandon.williams][~tiagomlalves]Thank you very much for your reply. Regarding the granularity of RIP, my main point is that it mainly depends on the needs of the user. My patch allows the user to select seconds, milliseconds and microseconds. All I need to do is to ensure that it is consistent with Cassandra 's timestamp itself. bq. What I wonder is, in which scenarios would microsecond-level PIT restore would be useful? [~brandon.williams] may have already described it clearly, but what I may want to say again is, if a batch of data is deleted by mistake, then accurate time granularity to microseconds is the only prerequisite to ensure that all data can be restored in c*. Milliseconds and seconds are not enough, they may lost data. bq.couldn't we detect automatically the granularity of PIT restore based on the value I will update the PR again, and [~brandon.williams][~tiagomlalves] If you are willing, can you be the reviewer :) > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Assignee: Maxwell Guo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832992#comment-17832992 ] Maxwell Guo edited comment on CASSANDRA-19448 at 4/2/24 2:20 AM: - [~brandon.williams][~tiagomlalves]Thank you very much for your reply. Regarding the granularity of RIP, my main point is that it mainly depends on the needs of the user. My patch allows the user to select seconds, milliseconds and microseconds. All I need to do is to ensure that it is consistent with Cassandra 's timestamp itself.C* provides timestamps up to microseconds, so users can restore to a specific piece of data. bq. What I wonder is, in which scenarios would microsecond-level PIT restore would be useful? [~brandon.williams] may have already described it clearly, but what I may want to say again is, if a batch of data is deleted by mistake, then accurate time granularity to microseconds is the only prerequisite to ensure that all data can be restored in c*. Milliseconds and seconds are not enough, they may lost data. bq.couldn't we detect automatically the granularity of PIT restore based on the value I will update the PR again, and [~brandon.williams][~tiagomlalves] If you are willing, can you be the reviewer :) was (Author: maxwellguo): [~brandon.williams][~tiagomlalves]Thank you very much for your reply. Regarding the granularity of RIP, my main point is that it mainly depends on the needs of the user. My patch allows the user to select seconds, milliseconds and microseconds. All I need to do is to ensure that it is consistent with Cassandra 's timestamp itself.As a said, c* provides timestamps up to microseconds, so users can restore to a specific piece of data. bq. What I wonder is, in which scenarios would microsecond-level PIT restore would be useful? [~brandon.williams] may have already described it clearly, but what I may want to say again is, if a batch of data is deleted by mistake, then accurate time granularity to microseconds is the only prerequisite to ensure that all data can be restored in c*. Milliseconds and seconds are not enough, they may lost data. bq.couldn't we detect automatically the granularity of PIT restore based on the value I will update the PR again, and [~brandon.williams][~tiagomlalves] If you are willing, can you be the reviewer :) > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Assignee: Maxwell Guo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832992#comment-17832992 ] Maxwell Guo edited comment on CASSANDRA-19448 at 4/2/24 2:20 AM: - [~brandon.williams][~tiagomlalves]Thank you very much for your reply. Regarding the granularity of RIP, my main point is that it mainly depends on the needs of the user. My patch allows the user to select seconds, milliseconds and microseconds. All I need to do is to ensure that it is consistent with Cassandra 's timestamp itself. bq. What I wonder is, in which scenarios would microsecond-level PIT restore would be useful? [~brandon.williams] may have already described it clearly, but what I may want to say again is, if a batch of data is deleted by mistake, then accurate time granularity to microseconds is the only prerequisite to ensure that all data can be restored in c*. Milliseconds and seconds are not enough, they may lost data. bq.couldn't we detect automatically the granularity of PIT restore based on the value I will update the PR again, and [~brandon.williams][~tiagomlalves] If you are willing, can you be the reviewer :) was (Author: maxwellguo): [~brandon.williams][~tiagomlalves]Thank you very much for your reply. Regarding the granularity of RIP, my main point is that it mainly depends on the needs of the user. My patch allows the user to select seconds, milliseconds and microseconds. All I need to do is to ensure that it is consistent with Cassandra 's timestamp itself.C* provides timestamps up to microseconds, so users can restore to a specific piece of data. bq. What I wonder is, in which scenarios would microsecond-level PIT restore would be useful? [~brandon.williams] may have already described it clearly, but what I may want to say again is, if a batch of data is deleted by mistake, then accurate time granularity to microseconds is the only prerequisite to ensure that all data can be restored in c*. Milliseconds and seconds are not enough, they may lost data. bq.couldn't we detect automatically the granularity of PIT restore based on the value I will update the PR again, and [~brandon.williams][~tiagomlalves] If you are willing, can you be the reviewer :) > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Assignee: Maxwell Guo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832369#comment-17832369 ] Maxwell Guo edited comment on CASSANDRA-19448 at 4/1/24 1:45 PM: - Hi Sorry for the late reply. I was in a hurry recently so I was away for a while. [~brandon.williams][~tiagomlalves] [~jeromatron], I have also made a PR for this ,And I might want a more general solution to this problem, whether rip is seconds, milliseconds, or microseconds. I have just prepare the pr but have not been tested very complete, here is the pr , and I will update the CI after they finished. Besides, I used to pay more attention to whether this category is a bug or an improvement, because it is related to whether to prepare a copy for 4.x. ||Heading 1||Heading 2|| |PR for trunk |[pr|https://github.com/apache/cassandra/pull/3215/]| And the ci are running ,will check if the they finished. ||Heading 1||Heading 2|| |java11|[java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/549/workflows/f1110dc1-b08e-4db5-97bf-e945658dc28b]| |java17|[java17|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/549/workflows/e9cf230a-9b78-4568-9ae4-14c0e68510a4]| Update: fix build error for unused imports was (Author: maxwellguo): Hi Sorry for the late reply. I was in a hurry recently so I was away for a while. [~brandon.williams][~tiagomlalves] [~jeromatron], I have also made a PR for this ,And I might want a more general solution to this problem, whether rip is seconds, milliseconds, or microseconds. I have just prepare the pr but have not been tested very complete, here is the pr , and I will update the CI after they finished. Besides, I used to pay more attention to whether this category is a bug or an improvement, because it is related to whether to prepare a copy for 4.x. ||Heading 1||Heading 2|| |PR for trunk |[pr|https://github.com/apache/cassandra/pull/3215/]| And the ci are running ,will check if the they finished. ||Heading 1||Heading 2|| |java11|[java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/547/workflows/8102b770-a623-4417-a5ea-5c602b7e6949]| |java17|[java17|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/547/workflows/82eb1c09-ca27-46b0-996c-7d841e062423]| > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Assignee: Maxwell Guo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832369#comment-17832369 ] Maxwell Guo edited comment on CASSANDRA-19448 at 3/30/24 7:23 AM: -- Hi Sorry for the late reply. I was in a hurry recently so I was away for a while. [~brandon.williams][~tiagomlalves] [~jeromatron], I have also made a PR for this ,And I might want a more general solution to this problem, whether rip is seconds, milliseconds, or microseconds. I have just prepare the pr but have not been tested very complete, here is the pr , and I will update the CI after they finished. Besides, I used to pay more attention to whether this category is a bug or an improvement, because it is related to whether to prepare a copy for 4.x. ||Heading 1||Heading 2|| |PR for trunk |[pr|https://github.com/apache/cassandra/pull/3215/]| And the ci are running ,will check if the they finished. ||Heading 1||Heading 2|| |java11|[java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/547/workflows/8102b770-a623-4417-a5ea-5c602b7e6949]| |java17|[java17|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/547/workflows/82eb1c09-ca27-46b0-996c-7d841e062423]| was (Author: maxwellguo): Hi [~brandon.williams][~tiagomlalves] [~jeromatron], I have also made a PR for this ,And I might want a more general solution to this problem, whether rip is seconds, milliseconds, or microseconds. I have just prepare the pr but have not been tested very complete, here is the pr , and I will update the CI after they finished. Besides, I used to pay more attention to whether this category is a bug or an improvement, because it is related to whether to prepare a copy for 4.x. ||Heading 1||Heading 2|| |PR for trunk |[pr|https://github.com/apache/cassandra/pull/3215/]| And the ci are running ,will check if the they finished. ||Heading 1||Heading 2|| |java11|[java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/547/workflows/8102b770-a623-4417-a5ea-5c602b7e6949]| |java17|[java17|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/547/workflows/82eb1c09-ca27-46b0-996c-7d841e062423]| > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Assignee: Maxwell Guo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832369#comment-17832369 ] Maxwell Guo edited comment on CASSANDRA-19448 at 3/30/24 7:16 AM: -- Hi [~brandon.williams][~tiagomlalves] [~jeromatron], I have also made a PR for this ,And I might want a more general solution to this problem, whether rip is seconds, milliseconds, or microseconds. I have just prepare the pr but have not been tested very complete, here is the pr , and I will update the CI after they finished. Besides, I used to pay more attention to whether this category is a bug or an improvement, because it is related to whether to prepare a copy for 4.x. ||Heading 1||Heading 2|| |PR for trunk |[pr|https://github.com/apache/cassandra/pull/3215/]| And the ci are running ,will check if the they finished. ||Heading 1||Heading 2|| |java11|[java11|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/547/workflows/8102b770-a623-4417-a5ea-5c602b7e6949]| |java17|[java17|https://app.circleci.com/pipelines/github/Maxwell-Guo/cassandra/547/workflows/82eb1c09-ca27-46b0-996c-7d841e062423]| was (Author: maxwellguo): Hi [~brandon.williams][~tiagomlalves] [~jeromatron], I have also made a PR for this ,And I might want a more general solution to this problem, whether rip is seconds, milliseconds, or microseconds. I have just prepare the pr but have not been tested very complete, here is the pr , and I will update the CI after they finished. Besides, I used to pay more attention to whether this category is a bug or an improvement, because it is related to whether to prepare a copy for 4.x. ||Heading 1||Heading 2|| |PR for trunk |https://github.com/apache/cassandra/pull/3215/| > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Assignee: Maxwell Guo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832369#comment-17832369 ] Maxwell Guo edited comment on CASSANDRA-19448 at 3/30/24 7:12 AM: -- Hi [~brandon.williams][~tiagomlalves] [~jeromatron], I have also made a PR for this ,And I might want a more general solution to this problem, whether rip is seconds, milliseconds, or microseconds. I have just prepare the pr but have not been tested very complete, here is the pr , and I will update the CI after they finished. Besides, I used to pay more attention to whether this category is a bug or an improvement, because it is related to whether to prepare a copy for 4.x. ||Heading 1||Heading 2|| |PR for trunk |https://github.com/apache/cassandra/pull/3215/| was (Author: maxwellguo): Hi [~brandon.williams][~tiagomlalves] [~jeromatron], I have also made a PR for this ,And I might want a more general solution to this problem, whether rip is seconds, milliseconds, or microseconds. I have just prepare the pr but have not been tested very complete, here is the pr , and I will update the CI after they finished. Besides, I used to pay more attention to whether this category is a bug or an improvement, because it is related to whether to prepare a copy for 4.x. > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Assignee: Maxwell Guo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822338#comment-17822338 ] Maxwell Guo edited comment on CASSANDRA-19448 at 3/1/24 2:16 AM: - ok，then let's treat it as a bug . was (Author: maxwellguo): ok，then let's left it as a bug . > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Assignee: Maxwell Guo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822210#comment-17822210 ] Jeremy Hanna edited comment on CASSANDRA-19448 at 2/29/24 4:28 PM: --- It's currently unassigned so feel free to take a look. Thanks! I've thought about it as a bug just because Cassandra stores update times as milliseconds or microseconds and there is nothing in the description that says that you can't use that granularity. It's just that the example is in seconds. It's not clear and there's no warning or error if you give it something with a granularity greater than seconds - it just ignores it. What to do about that could be either to: # be clearer in the docs and have a warning/error when users try to use a granularity greater than seconds. # make it respect greater granularities which aligns more with the C* write timestamp formats I think 2 is the better outcome. So I think it could be argued as a bug or an improvement. [~brandon.williams] do you have any thoughts on bug or improvement designation? was (Author: jeromatron): It's currently unassigned so feel free to take a look. Thanks! I've thought about it as a bug just because Cassandra stores update times as milliseconds or microseconds and there is nothing in the description that says that you can't use that granularity. It's just that the example is in seconds. Since it's not clear and there's no warning or error if you give it something with a granularity greater than seconds - it just ignores it. What to do about that could be either to: # be clearer in the docs and have a warning/error when users try to use a granularity greater than seconds. # make it respect greater granularities which aligns more with the C* write timestamp formats I think 2 is the better outcome. So I think it could be argued as a bug or an improvement. [~brandon.williams] do you have any thoughts on bug or improvement designation? > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-19448) CommitlogArchiver only has granularity to seconds for restore_point_in_time
[ https://issues.apache.org/jira/browse/CASSANDRA-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821988#comment-17821988 ] Maxwell Guo edited comment on CASSANDRA-19448 at 2/29/24 7:58 AM: -- I don't think this is a bug. This just can ensure that cassandra can ensure the rpo to second level, as the description [here|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties#L39] for restore_point_in_time do not say that we guarantee the rpo can be microseconds level or some other level. I think this issue can be marked as an improvement. And I think the requirements raised by this issue are very meaningful.As I also encountered this problem when doing backup and restore for cassandra before. If this issue is unassigned ,I think I may do some help. :D was (Author: maxwellguo): I don't think this is a bug. This just can ensure that cassandra can ensure the rpo to second level, as the description [here|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties#L39] for restore_point_in_time do not say that we guarantee the rpo can be microseconds level or some other level. I think this issue can be marked as an improvement. And I think the requirements raised by this issue are very meaningful.As I also encountered this problem when doing backup and restore for cassandra before. If this issue is not assigned ,I think I may do some help. :D > CommitlogArchiver only has granularity to seconds for restore_point_in_time > --- > > Key: CASSANDRA-19448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19448 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jeremy Hanna >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > > Commitlog archiver allows users to backup commitlog files for the purpose of > doing point in time restores. The [configuration > file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] > gives an example of down to the seconds granularity but then asks what > whether the timestamps are microseconds or milliseconds - defaulting to > microseconds. Because the [CommitLogArchiver uses a second based date > format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], > if a user specifies to restore at something at a lower granularity like > milliseconds or microseconds, that means that the it will truncate everything > after the second and restore to that second. So say you specify a > restore_point_in_time like this: > restore_point_in_time=2024:01:18 17:01:01.623392 > it will silently truncate everything after the 01 seconds. So effectively to > the user, it is missing updates between 01 and 01.623392. > This appears to be a bug in the intent. We should allow users to specify > down to the millisecond or even microsecond level. If we allow them to > specify down to microseconds for the restore point in time, then it may > internally need to change from a long. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
