[
https://issues.apache.org/jira/browse/CASSANDRA-20829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18014067#comment-18014067
]
Stefan Miklosovic edited comment on CASSANDRA-20829 at 8/15/25 8:20 AM:
------------------------------------------------------------------------
[CASSANDRA-20829-4.0|https://github.com/instaclustr/cassandra/tree/CASSANDRA-20829-4.0]
{noformat}
java11_pre-commit_tests
✓ j11_build 1m 46s
✓ j11_cqlsh-dtests-py2-no-vnodes 5m 25s
✓ j11_cqlsh-dtests-py2-with-vnodes 5m 44s
✓ j11_cqlsh_dtests_py3 5m 20s
✓ j11_cqlsh_dtests_py311 5m 46s
✓ j11_cqlsh_dtests_py311_vnode 5m 30s
✓ j11_cqlsh_dtests_py38 5m 37s
✓ j11_cqlsh_dtests_py38_vnode 6m 0s
✓ j11_cqlsh_dtests_py3_vnode 5m 59s
✓ j11_cqlshlib_tests 9m 4s
✓ j11_dtests_vnode 43m 48s
✓ j11_jvm_dtests 13m 46s
✓ j11_unit_tests 7m 58s
✕ j11_dtests 55m 5s
refresh_test.TestRefresh test_refresh_deadlock_startup
✕ j11_unit_tests_repeat 31m 43s
org.apache.cassandra.index.CustomIndexTest testFailing2iFlush
java8_pre-commit_tests
✓ j8_build 2m 31s
✓ j8_cqlsh-dtests-py2-no-vnodes 5m 35s
✓ j8_cqlsh-dtests-py2-with-vnodes 6m 39s
✓ j8_cqlsh_dtests_py3 5m 22s
✓ j8_cqlsh_dtests_py38 6m 25s
✓ j8_cqlsh_dtests_py38_vnode 7m 7s
✓ j8_cqlsh_dtests_py3_vnode 6m 21s
✓ j8_cqlshlib_tests 9m 51s
✓ j8_dtests_vnode 42m 9s
✓ j8_jvm_dtests 15m 14s
✓ j8_unit_tests 7m 50s
✓ j8_utests_system_keyspace_directory 7m 46s
✓ j8_utests_system_keyspace_directory_repeat 30m 58s
✓ j11_unit_tests_repeat 30m 28s
✓ j11_unit_tests 8m 35s
✓ j11_dtests_vnode 42m 43s
✓ j11_cqlsh_dtests_py3_vnode 5m 25s
✓ j11_cqlsh_dtests_py38_vnode 5m 45s
✓ j11_cqlsh_dtests_py38 5m 22s
✓ j11_cqlsh_dtests_py311_vnode 5m 54s
✓ j11_cqlsh_dtests_py311 5m 28s
✓ j11_cqlsh_dtests_py3 5m 42s
✓ j11_cqlsh-dtests-py2-with-vnodes 5m 48s
✓ j11_cqlsh-dtests-py2-no-vnodes 5m 21s
✕ j8_dtests 54m 47s
refresh_test.TestRefresh test_refresh_deadlock_startup
✕ j8_unit_tests_repeat 31m 19s
org.apache.cassandra.index.CustomIndexTest testFailing2iFlush
✕ j11_dtests 54m 50s
refresh_test.TestRefresh test_refresh_deadlock_startup
{noformat}[java11_pre-commit_tests|https://app.circleci.com/pipelines/github/instaclustr/cassandra/5939/workflows/51d1a5a2-3338-4def-9437-f7fc01d441ce]
[java8_pre-commit_tests|https://app.circleci.com/pipelines/github/instaclustr/cassandra/5939/workflows/80f1331a-4621-4885-b256-2993974989c6]
org.apache.cassandra.index.CustomIndexTest testFailing2iFlush is just flaky and
multiplexer just reveiled that.
was (Author: smiklosovic):
{code:java}
[CASSANDRA-20829-4.0|https://github.com/instaclustr/cassandra/tree/CASSANDRA-20829-4.0]
{noformat}
java11_pre-commit_tests
✓ j11_build 1m 46s
✓ j11_cqlsh-dtests-py2-no-vnodes 5m 25s
✓ j11_cqlsh-dtests-py2-with-vnodes 5m 44s
✓ j11_cqlsh_dtests_py3 5m 20s
✓ j11_cqlsh_dtests_py311 5m 46s
✓ j11_cqlsh_dtests_py311_vnode 5m 30s
✓ j11_cqlsh_dtests_py38 5m 37s
✓ j11_cqlsh_dtests_py38_vnode 6m 0s
✓ j11_cqlsh_dtests_py3_vnode 5m 59s
✓ j11_cqlshlib_tests 9m 4s
✓ j11_dtests_vnode 43m 48s
✓ j11_jvm_dtests 13m 46s
✓ j11_unit_tests 7m 58s
✕ j11_dtests 55m 5s
refresh_test.TestRefresh test_refresh_deadlock_startup
✕ j11_unit_tests_repeat 31m 43s
org.apache.cassandra.index.CustomIndexTest testFailing2iFlush
java8_pre-commit_tests
✓ j8_build 2m 31s
✓ j8_cqlsh-dtests-py2-no-vnodes 5m 35s
✓ j8_cqlsh-dtests-py2-with-vnodes 6m 39s
✓ j8_cqlsh_dtests_py3 5m 22s
✓ j8_cqlsh_dtests_py38 6m 25s
✓ j8_cqlsh_dtests_py38_vnode 7m 7s
✓ j8_cqlsh_dtests_py3_vnode 6m 21s
✓ j8_cqlshlib_tests 9m 51s
✓ j8_dtests_vnode 42m 9s
✓ j8_jvm_dtests 15m 14s
✓ j8_unit_tests 7m 50s
✓ j8_utests_system_keyspace_directory 7m 46s
✓ j8_utests_system_keyspace_directory_repeat 30m 58s
✓ j11_unit_tests_repeat 30m 28s
✓ j11_unit_tests 8m 35s
✓ j11_dtests_vnode 42m 43s
✓ j11_cqlsh_dtests_py3_vnode 5m 25s
✓ j11_cqlsh_dtests_py38_vnode 5m 45s
✓ j11_cqlsh_dtests_py38 5m 22s
✓ j11_cqlsh_dtests_py311_vnode 5m 54s
✓ j11_cqlsh_dtests_py311 5m 28s
✓ j11_cqlsh_dtests_py3 5m 42s
✓ j11_cqlsh-dtests-py2-with-vnodes 5m 48s
✓ j11_cqlsh-dtests-py2-no-vnodes 5m 21s
✕ j8_dtests 54m 47s
refresh_test.TestRefresh test_refresh_deadlock_startup
✕ j8_unit_tests_repeat 31m 19s
org.apache.cassandra.index.CustomIndexTest testFailing2iFlush
✕ j11_dtests 54m 50s
refresh_test.TestRefresh test_refresh_deadlock_startup
{noformat}[java11_pre-commit_tests|https://app.circleci.com/pipelines/github/instaclustr/cassandra/5939/workflows/51d1a5a2-3338-4def-9437-f7fc01d441ce]
[java8_pre-commit_tests|https://app.circleci.com/pipelines/github/instaclustr/cassandra/5939/workflows/80f1331a-4621-4885-b256-2993974989c6]{code}
org.apache.cassandra.index.CustomIndexTest testFailing2iFlush is just flaky and
multiplexer just reveiled that.
> Secondary index implementations do not integrate with IndexGCTransaction when
> compaction contains fully expired SSTables
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-20829
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20829
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Feature/2i Index, Local/Compaction, Local/Compaction/TWCS
> Reporter: Stefan Miklosovic
> Assignee: Stefan Miklosovic
> Priority: Normal
> Fix For: 4.0.x, 4.1.x
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> There is a test (1) which ensures that when data are TTLed and compacted,
> IndexGCTransaction is aware of that and it will invoke Indexer.removeRow()
> method eventually.
> However, this is not working properly when we have fully expired SSTables,
> e.g. as the result of a table being on TWCS and having TTL on that.
> The reason is that in CompactionTask, we are filtering out fully expired ones
> (2). These then do not go to the compaction process and then they are not
> reacted on in listener() (3) which contains this logic (4). Eventually,
> onRowMerge in IndexGCTransaction will make the diff and in its commit
> indexer.removeRow(row); will notify 2i about its removal.
>
> This integration is missing and it is quite a big problem because if there
> are custom secondary index implementations the fact that SSTables were fully
> expired is not propagated to them which means that data are never removed
> from whatever backend they use.
> The solution is to go to the compaction with fully expired SSTables as well
> _but only if we detected that respective column family has some indexes_
>
> (1)
> [https://github.com/apache/cassandra/blob/cassandra-4.1/test/unit/org/apache/cassandra/index/CustomIndexTest.java#L583-L607]
> (2)
> [https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/db/compaction/CompactionTask.java#L174]
> (3)
> [https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/db/compaction/CompactionIterator.java#L130]
> (4)
> [https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/db/compaction/CompactionIterator.java#L235-L252]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
