[ https://issues.apache.org/jira/browse/CASSANDRA-16547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17311623#comment-17311623 ]
Benjamin Lerer edited comment on CASSANDRA-16547 at 3/30/21, 4:05 PM: ---------------------------------------------------------------------- The problem comes from the fact that {{SizeTieredCompactionStrategy}} and {{TimeWindowCompactionStrategy}} use a cached value for {{getEstimatedRemainingTasks()}}. That value is only recomputed when {{getNextBackgroundTask}} is called. As {{getEstimatedRemainingTasks}} is used to prioritize the compactions before {{getNextBackgroundTask}} is called, the prioritisation might rely on outdated numbers. As the {{pendingTasks}} and {{pendingTasksByTableName}} compaction metrics rely also on {{getEstimatedRemainingTasks}} they will also provide an outdated view of the compaction state. Making {{getEstimatedRemainingTasks()}} compute the estimates on each call is probalby not a good option as that computation will be triggered each time that the metrics are fetched. My proposal would be to recompute the cached estimates on SSTables changes. The estimate will not be fully accurate but it will be closer from reallity. [~marcuse] Does that approach makes sense to you? It also seems that this problem is not new. {{3.11}} suffer from the same issue if I am not mistaken. By consequence it is unclear to me if we should consider that ticket as a bloquer for 4.0 GA. After looking into the code it seems that we have this problem in the previous versions too. was (Author: blerer): The problem comes from the fact that {{SizeTieredCompactionStrategy}} and {{TimeWindowCompactionStrategy}} use a cached value for {{getEstimatedRemainingTasks()}}. That value is only recomputed when {{getNextBackgroundTask}} is called. As {{getEstimatedRemainingTasks}} is used to prioritize the compactions before {{getNextBackgroundTask}} is called, the prioritisation might rely on outdated numbers. As the {{pendingTasks}} and {{pendingTasksByTableName}} compaction metrics relies on {{getEstimatedRemainingTasks}} they will also provide an outdated view of the compaction state. Making {{getEstimatedRemainingTasks()}} compute the estimates on each call is probalby not a good option as that computation will be triggered each time that the metrics are fetched. My proposal would be to recompute the cached estimates on SSTables changes. The estimate will not be fully accurate but it will be closer from reallity. [~marcuse] Does that approach makes sense to you? It also seems that this problem is not new. {{3.11}} suffer from the same issue if I am not mistaken. By consequence it is unclear to me if we should consider that ticket as a bloquer for 4.0 GA. After looking into the code it seems that we have this problem in the previous versions too. > Prioritisation for sized-tier and TW compactions is based on outdated > estimation > -------------------------------------------------------------------------------- > > Key: CASSANDRA-16547 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16547 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction > Reporter: Benjamin Lerer > Assignee: Benjamin Lerer > Priority: Normal > Fix For: 4.0-rc > > > Due to the way getEstimatedRemainingTasks() works, it looks that compactions > prioritisation are based on outdated estimations (see > https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/compaction/CompactionStrategyHolder.java#L109). > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org