[jira] [Updated] (CASSANDRA-18260) Add details to Error message: Not enough space for compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-18260: --- Fix Version/s: 4.0.10 4.1.2 5.0 (was: 5.x) (was: 4.0.x) (was: 4.1.x) Source Control Link: https://github.com/apache/cassandra/commit/65c99bfc42015ccfbcbc8963b3fb3c2a761ea053 Resolution: Fixed Status: Resolved (was: Ready to Commit) Committed as [65c99bfc42015ccfbcbc8963b3fb3c2a761ea053 |https://github.com/apache/cassandra/commit/65c99bfc42015ccfbcbc8963b3fb3c2a761ea053]. > Add details to Error message: Not enough space for compaction > -- > > Key: CASSANDRA-18260 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18260 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction, Observability/Logging >Reporter: Brad Schoening >Assignee: Henrik Ingo >Priority: Normal > Fix For: 4.0.10, 4.1.2, 5.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > When compaction fails, the log message should list a) the free space > available on disk at that point in time and b) perhaps the number and/or size > of the source sstables being compacted. > Free space can change from one moment to the next, so when the below > compaction failed because it needed 161GB, upon checking the server a few > minutes later, it had 184GB free. Similarly, the error message mentions it > was writing out one SSTable on this STCS table, but its not clear if it was > combining X -> 1 tables, or something else. > {noformat} > [INFO ] [CompactionExecutor:77758] cluster_id=87 ip_address=127.1.1.1 > CompactionTask.java:241 - Compacted (8a1cffe0-abb5-11ed-b3fc-8d2ac2c52f0d) 1 > sstables to [...] to level=0. 86.997GiB to 86.997GiB (~99% of original) in > 1,508,155ms. Read Throughput = 59.069MiB/s, Write Throughput = 59.069MiB/s, > Row Throughput = ~20,283/s. 21,375 total partitions merged to 21,375. > Partition merge counts were \{1:21375, } > [ERROR] [CompactionExecutor:4] cluster_id=87 ip_address=127.1.1.1 > CassandraDaemon.java:581 - Exception in thread > Thread[CompactionExecutor:4,1,main] > java.lang.RuntimeException: Not enough space for compaction, estimated > sstables = 1, expected write size = 161228934093 > at > org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100) > at > org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613) > at > org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:834) > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18260) Add details to Error message: Not enough space for compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-18260: --- Status: Review In Progress (was: Needs Committer) > Add details to Error message: Not enough space for compaction > -- > > Key: CASSANDRA-18260 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18260 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction, Observability/Logging >Reporter: Brad Schoening >Assignee: Henrik Ingo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.x > > Time Spent: 3h 20m > Remaining Estimate: 0h > > When compaction fails, the log message should list a) the free space > available on disk at that point in time and b) perhaps the number and/or size > of the source sstables being compacted. > Free space can change from one moment to the next, so when the below > compaction failed because it needed 161GB, upon checking the server a few > minutes later, it had 184GB free. Similarly, the error message mentions it > was writing out one SSTable on this STCS table, but its not clear if it was > combining X -> 1 tables, or something else. > {noformat} > [INFO ] [CompactionExecutor:77758] cluster_id=87 ip_address=127.1.1.1 > CompactionTask.java:241 - Compacted (8a1cffe0-abb5-11ed-b3fc-8d2ac2c52f0d) 1 > sstables to [...] to level=0. 86.997GiB to 86.997GiB (~99% of original) in > 1,508,155ms. Read Throughput = 59.069MiB/s, Write Throughput = 59.069MiB/s, > Row Throughput = ~20,283/s. 21,375 total partitions merged to 21,375. > Partition merge counts were \{1:21375, } > [ERROR] [CompactionExecutor:4] cluster_id=87 ip_address=127.1.1.1 > CassandraDaemon.java:581 - Exception in thread > Thread[CompactionExecutor:4,1,main] > java.lang.RuntimeException: Not enough space for compaction, estimated > sstables = 1, expected write size = 161228934093 > at > org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100) > at > org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613) > at > org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:834) > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18260) Add details to Error message: Not enough space for compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-18260: --- Status: Ready to Commit (was: Review In Progress) > Add details to Error message: Not enough space for compaction > -- > > Key: CASSANDRA-18260 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18260 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction, Observability/Logging >Reporter: Brad Schoening >Assignee: Henrik Ingo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.x > > Time Spent: 3h 20m > Remaining Estimate: 0h > > When compaction fails, the log message should list a) the free space > available on disk at that point in time and b) perhaps the number and/or size > of the source sstables being compacted. > Free space can change from one moment to the next, so when the below > compaction failed because it needed 161GB, upon checking the server a few > minutes later, it had 184GB free. Similarly, the error message mentions it > was writing out one SSTable on this STCS table, but its not clear if it was > combining X -> 1 tables, or something else. > {noformat} > [INFO ] [CompactionExecutor:77758] cluster_id=87 ip_address=127.1.1.1 > CompactionTask.java:241 - Compacted (8a1cffe0-abb5-11ed-b3fc-8d2ac2c52f0d) 1 > sstables to [...] to level=0. 86.997GiB to 86.997GiB (~99% of original) in > 1,508,155ms. Read Throughput = 59.069MiB/s, Write Throughput = 59.069MiB/s, > Row Throughput = ~20,283/s. 21,375 total partitions merged to 21,375. > Partition merge counts were \{1:21375, } > [ERROR] [CompactionExecutor:4] cluster_id=87 ip_address=127.1.1.1 > CassandraDaemon.java:581 - Exception in thread > Thread[CompactionExecutor:4,1,main] > java.lang.RuntimeException: Not enough space for compaction, estimated > sstables = 1, expected write size = 161228934093 > at > org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100) > at > org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613) > at > org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:834) > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18260) Add details to Error message: Not enough space for compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maxwellguo updated CASSANDRA-18260: --- Status: Needs Committer (was: Review In Progress) > Add details to Error message: Not enough space for compaction > -- > > Key: CASSANDRA-18260 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18260 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction, Observability/Logging >Reporter: Brad Schoening >Assignee: Henrik Ingo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.x > > Time Spent: 3h 20m > Remaining Estimate: 0h > > When compaction fails, the log message should list a) the free space > available on disk at that point in time and b) perhaps the number and/or size > of the source sstables being compacted. > Free space can change from one moment to the next, so when the below > compaction failed because it needed 161GB, upon checking the server a few > minutes later, it had 184GB free. Similarly, the error message mentions it > was writing out one SSTable on this STCS table, but its not clear if it was > combining X -> 1 tables, or something else. > {noformat} > [INFO ] [CompactionExecutor:77758] cluster_id=87 ip_address=127.1.1.1 > CompactionTask.java:241 - Compacted (8a1cffe0-abb5-11ed-b3fc-8d2ac2c52f0d) 1 > sstables to [...] to level=0. 86.997GiB to 86.997GiB (~99% of original) in > 1,508,155ms. Read Throughput = 59.069MiB/s, Write Throughput = 59.069MiB/s, > Row Throughput = ~20,283/s. 21,375 total partitions merged to 21,375. > Partition merge counts were \{1:21375, } > [ERROR] [CompactionExecutor:4] cluster_id=87 ip_address=127.1.1.1 > CassandraDaemon.java:581 - Exception in thread > Thread[CompactionExecutor:4,1,main] > java.lang.RuntimeException: Not enough space for compaction, estimated > sstables = 1, expected write size = 161228934093 > at > org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100) > at > org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613) > at > org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:834) > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18260) Add details to Error message: Not enough space for compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-18260: --- Fix Version/s: 5.x > Add details to Error message: Not enough space for compaction > -- > > Key: CASSANDRA-18260 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18260 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction, Observability/Logging >Reporter: Brad Schoening >Assignee: Henrik Ingo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.x > > Time Spent: 1h 40m > Remaining Estimate: 0h > > When compaction fails, the log message should list a) the free space > available on disk at that point in time and b) perhaps the number and/or size > of the source sstables being compacted. > Free space can change from one moment to the next, so when the below > compaction failed because it needed 161GB, upon checking the server a few > minutes later, it had 184GB free. Similarly, the error message mentions it > was writing out one SSTable on this STCS table, but its not clear if it was > combining X -> 1 tables, or something else. > {noformat} > [INFO ] [CompactionExecutor:77758] cluster_id=87 ip_address=127.1.1.1 > CompactionTask.java:241 - Compacted (8a1cffe0-abb5-11ed-b3fc-8d2ac2c52f0d) 1 > sstables to [...] to level=0. 86.997GiB to 86.997GiB (~99% of original) in > 1,508,155ms. Read Throughput = 59.069MiB/s, Write Throughput = 59.069MiB/s, > Row Throughput = ~20,283/s. 21,375 total partitions merged to 21,375. > Partition merge counts were \{1:21375, } > [ERROR] [CompactionExecutor:4] cluster_id=87 ip_address=127.1.1.1 > CassandraDaemon.java:581 - Exception in thread > Thread[CompactionExecutor:4,1,main] > java.lang.RuntimeException: Not enough space for compaction, estimated > sstables = 1, expected write size = 161228934093 > at > org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100) > at > org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613) > at > org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:834) > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18260) Add details to Error message: Not enough space for compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maxwellguo updated CASSANDRA-18260: --- Fix Version/s: (was: 5.x) > Add details to Error message: Not enough space for compaction > -- > > Key: CASSANDRA-18260 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18260 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction, Observability/Logging >Reporter: Brad Schoening >Assignee: Henrik Ingo >Priority: Normal > Fix For: 4.0.x, 4.1.x > > Time Spent: 1h 40m > Remaining Estimate: 0h > > When compaction fails, the log message should list a) the free space > available on disk at that point in time and b) perhaps the number and/or size > of the source sstables being compacted. > Free space can change from one moment to the next, so when the below > compaction failed because it needed 161GB, upon checking the server a few > minutes later, it had 184GB free. Similarly, the error message mentions it > was writing out one SSTable on this STCS table, but its not clear if it was > combining X -> 1 tables, or something else. > {noformat} > [INFO ] [CompactionExecutor:77758] cluster_id=87 ip_address=127.1.1.1 > CompactionTask.java:241 - Compacted (8a1cffe0-abb5-11ed-b3fc-8d2ac2c52f0d) 1 > sstables to [...] to level=0. 86.997GiB to 86.997GiB (~99% of original) in > 1,508,155ms. Read Throughput = 59.069MiB/s, Write Throughput = 59.069MiB/s, > Row Throughput = ~20,283/s. 21,375 total partitions merged to 21,375. > Partition merge counts were \{1:21375, } > [ERROR] [CompactionExecutor:4] cluster_id=87 ip_address=127.1.1.1 > CassandraDaemon.java:581 - Exception in thread > Thread[CompactionExecutor:4,1,main] > java.lang.RuntimeException: Not enough space for compaction, estimated > sstables = 1, expected write size = 161228934093 > at > org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100) > at > org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613) > at > org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:834) > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18260) Add details to Error message: Not enough space for compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maxwellguo updated CASSANDRA-18260: --- Reviewers: Brad Schoening, Claude Warren, Kan Maung, maxwellguo, Michael Semb Wever (was: Brad Schoening, Claude Warren, Kan Maung, Michael Semb Wever) > Add details to Error message: Not enough space for compaction > -- > > Key: CASSANDRA-18260 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18260 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction, Observability/Logging >Reporter: Brad Schoening >Assignee: Henrik Ingo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.x > > Time Spent: 1h 40m > Remaining Estimate: 0h > > When compaction fails, the log message should list a) the free space > available on disk at that point in time and b) perhaps the number and/or size > of the source sstables being compacted. > Free space can change from one moment to the next, so when the below > compaction failed because it needed 161GB, upon checking the server a few > minutes later, it had 184GB free. Similarly, the error message mentions it > was writing out one SSTable on this STCS table, but its not clear if it was > combining X -> 1 tables, or something else. > {noformat} > [INFO ] [CompactionExecutor:77758] cluster_id=87 ip_address=127.1.1.1 > CompactionTask.java:241 - Compacted (8a1cffe0-abb5-11ed-b3fc-8d2ac2c52f0d) 1 > sstables to [...] to level=0. 86.997GiB to 86.997GiB (~99% of original) in > 1,508,155ms. Read Throughput = 59.069MiB/s, Write Throughput = 59.069MiB/s, > Row Throughput = ~20,283/s. 21,375 total partitions merged to 21,375. > Partition merge counts were \{1:21375, } > [ERROR] [CompactionExecutor:4] cluster_id=87 ip_address=127.1.1.1 > CassandraDaemon.java:581 - Exception in thread > Thread[CompactionExecutor:4,1,main] > java.lang.RuntimeException: Not enough space for compaction, estimated > sstables = 1, expected write size = 161228934093 > at > org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100) > at > org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613) > at > org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:834) > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18260) Add details to Error message: Not enough space for compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henrik Ingo updated CASSANDRA-18260: Mentor: Michael Semb Wever Status: Review In Progress (was: Changes Suggested) > Add details to Error message: Not enough space for compaction > -- > > Key: CASSANDRA-18260 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18260 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction, Observability/Logging >Reporter: Brad Schoening >Assignee: Henrik Ingo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.x > > Time Spent: 1h > Remaining Estimate: 0h > > When compaction fails, the log message should list a) the free space > available on disk at that point in time and b) perhaps the number and/or size > of the source sstables being compacted. > Free space can change from one moment to the next, so when the below > compaction failed because it needed 161GB, upon checking the server a few > minutes later, it had 184GB free. Similarly, the error message mentions it > was writing out one SSTable on this STCS table, but its not clear if it was > combining X -> 1 tables, or something else. > {noformat} > [INFO ] [CompactionExecutor:77758] cluster_id=87 ip_address=127.1.1.1 > CompactionTask.java:241 - Compacted (8a1cffe0-abb5-11ed-b3fc-8d2ac2c52f0d) 1 > sstables to [...] to level=0. 86.997GiB to 86.997GiB (~99% of original) in > 1,508,155ms. Read Throughput = 59.069MiB/s, Write Throughput = 59.069MiB/s, > Row Throughput = ~20,283/s. 21,375 total partitions merged to 21,375. > Partition merge counts were \{1:21375, } > [ERROR] [CompactionExecutor:4] cluster_id=87 ip_address=127.1.1.1 > CassandraDaemon.java:581 - Exception in thread > Thread[CompactionExecutor:4,1,main] > java.lang.RuntimeException: Not enough space for compaction, estimated > sstables = 1, expected write size = 161228934093 > at > org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100) > at > org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613) > at > org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:834) > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18260) Add details to Error message: Not enough space for compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-18260: --- Status: Changes Suggested (was: Ready to Commit) [~henrik.ingo], the patch doesn't apply without conflict to 4.0 and 4.1 branches. Can I ask of you to create patches for both (or just one if it cleanly cherry-picks to the other). > Add details to Error message: Not enough space for compaction > -- > > Key: CASSANDRA-18260 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18260 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction, Observability/Logging >Reporter: Brad Schoening >Assignee: Henrik Ingo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.x > > Time Spent: 40m > Remaining Estimate: 0h > > When compaction fails, the log message should list a) the free space > available on disk at that point in time and b) perhaps the number and/or size > of the source sstables being compacted. > Free space can change from one moment to the next, so when the below > compaction failed because it needed 161GB, upon checking the server a few > minutes later, it had 184GB free. Similarly, the error message mentions it > was writing out one SSTable on this STCS table, but its not clear if it was > combining X -> 1 tables, or something else. > {noformat} > [INFO ] [CompactionExecutor:77758] cluster_id=87 ip_address=127.1.1.1 > CompactionTask.java:241 - Compacted (8a1cffe0-abb5-11ed-b3fc-8d2ac2c52f0d) 1 > sstables to [...] to level=0. 86.997GiB to 86.997GiB (~99% of original) in > 1,508,155ms. Read Throughput = 59.069MiB/s, Write Throughput = 59.069MiB/s, > Row Throughput = ~20,283/s. 21,375 total partitions merged to 21,375. > Partition merge counts were \{1:21375, } > [ERROR] [CompactionExecutor:4] cluster_id=87 ip_address=127.1.1.1 > CassandraDaemon.java:581 - Exception in thread > Thread[CompactionExecutor:4,1,main] > java.lang.RuntimeException: Not enough space for compaction, estimated > sstables = 1, expected write size = 161228934093 > at > org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100) > at > org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613) > at > org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:834) > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18260) Add details to Error message: Not enough space for compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-18260: --- Status: Ready to Commit (was: Review In Progress) > Add details to Error message: Not enough space for compaction > -- > > Key: CASSANDRA-18260 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18260 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction, Observability/Logging >Reporter: Brad Schoening >Assignee: Henrik Ingo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.x > > Time Spent: 40m > Remaining Estimate: 0h > > When compaction fails, the log message should list a) the free space > available on disk at that point in time and b) perhaps the number and/or size > of the source sstables being compacted. > Free space can change from one moment to the next, so when the below > compaction failed because it needed 161GB, upon checking the server a few > minutes later, it had 184GB free. Similarly, the error message mentions it > was writing out one SSTable on this STCS table, but its not clear if it was > combining X -> 1 tables, or something else. > {noformat} > [INFO ] [CompactionExecutor:77758] cluster_id=87 ip_address=127.1.1.1 > CompactionTask.java:241 - Compacted (8a1cffe0-abb5-11ed-b3fc-8d2ac2c52f0d) 1 > sstables to [...] to level=0. 86.997GiB to 86.997GiB (~99% of original) in > 1,508,155ms. Read Throughput = 59.069MiB/s, Write Throughput = 59.069MiB/s, > Row Throughput = ~20,283/s. 21,375 total partitions merged to 21,375. > Partition merge counts were \{1:21375, } > [ERROR] [CompactionExecutor:4] cluster_id=87 ip_address=127.1.1.1 > CassandraDaemon.java:581 - Exception in thread > Thread[CompactionExecutor:4,1,main] > java.lang.RuntimeException: Not enough space for compaction, estimated > sstables = 1, expected write size = 161228934093 > at > org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100) > at > org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613) > at > org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:834) > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18260) Add details to Error message: Not enough space for compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-18260: --- Reviewers: Brad Schoening, Claude Warren, Kan Maung, Michael Semb Wever (was: Claude Warren, Michael Semb Wever) > Add details to Error message: Not enough space for compaction > -- > > Key: CASSANDRA-18260 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18260 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction, Observability/Logging >Reporter: Brad Schoening >Assignee: Henrik Ingo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.x > > Time Spent: 40m > Remaining Estimate: 0h > > When compaction fails, the log message should list a) the free space > available on disk at that point in time and b) perhaps the number and/or size > of the source sstables being compacted. > Free space can change from one moment to the next, so when the below > compaction failed because it needed 161GB, upon checking the server a few > minutes later, it had 184GB free. Similarly, the error message mentions it > was writing out one SSTable on this STCS table, but its not clear if it was > combining X -> 1 tables, or something else. > {noformat} > [INFO ] [CompactionExecutor:77758] cluster_id=87 ip_address=127.1.1.1 > CompactionTask.java:241 - Compacted (8a1cffe0-abb5-11ed-b3fc-8d2ac2c52f0d) 1 > sstables to [...] to level=0. 86.997GiB to 86.997GiB (~99% of original) in > 1,508,155ms. Read Throughput = 59.069MiB/s, Write Throughput = 59.069MiB/s, > Row Throughput = ~20,283/s. 21,375 total partitions merged to 21,375. > Partition merge counts were \{1:21375, } > [ERROR] [CompactionExecutor:4] cluster_id=87 ip_address=127.1.1.1 > CassandraDaemon.java:581 - Exception in thread > Thread[CompactionExecutor:4,1,main] > java.lang.RuntimeException: Not enough space for compaction, estimated > sstables = 1, expected write size = 161228934093 > at > org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100) > at > org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613) > at > org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:834) > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18260) Add details to Error message: Not enough space for compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-18260: --- Reviewers: Claude Warren, Michael Semb Wever Status: Review In Progress (was: Patch Available) > Add details to Error message: Not enough space for compaction > -- > > Key: CASSANDRA-18260 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18260 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction, Observability/Logging >Reporter: Brad Schoening >Assignee: Henrik Ingo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.x > > Time Spent: 40m > Remaining Estimate: 0h > > When compaction fails, the log message should list a) the free space > available on disk at that point in time and b) perhaps the number and/or size > of the source sstables being compacted. > Free space can change from one moment to the next, so when the below > compaction failed because it needed 161GB, upon checking the server a few > minutes later, it had 184GB free. Similarly, the error message mentions it > was writing out one SSTable on this STCS table, but its not clear if it was > combining X -> 1 tables, or something else. > {noformat} > [INFO ] [CompactionExecutor:77758] cluster_id=87 ip_address=127.1.1.1 > CompactionTask.java:241 - Compacted (8a1cffe0-abb5-11ed-b3fc-8d2ac2c52f0d) 1 > sstables to [...] to level=0. 86.997GiB to 86.997GiB (~99% of original) in > 1,508,155ms. Read Throughput = 59.069MiB/s, Write Throughput = 59.069MiB/s, > Row Throughput = ~20,283/s. 21,375 total partitions merged to 21,375. > Partition merge counts were \{1:21375, } > [ERROR] [CompactionExecutor:4] cluster_id=87 ip_address=127.1.1.1 > CassandraDaemon.java:581 - Exception in thread > Thread[CompactionExecutor:4,1,main] > java.lang.RuntimeException: Not enough space for compaction, estimated > sstables = 1, expected write size = 161228934093 > at > org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100) > at > org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613) > at > org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:834) > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18260) Add details to Error message: Not enough space for compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-18260: --- Component/s: Local/Compaction > Add details to Error message: Not enough space for compaction > -- > > Key: CASSANDRA-18260 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18260 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction, Observability/Logging >Reporter: Brad Schoening >Assignee: Henrik Ingo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > When compaction fails, the log message should list a) the free space > available on disk at that point in time and b) perhaps the number and/or size > of the source sstables being compacted. > Free space can change from one moment to the next, so when the below > compaction failed because it needed 161GB, upon checking the server a few > minutes later, it had 184GB free. Similarly, the error message mentions it > was writing out one SSTable on this STCS table, but its not clear if it was > combining X -> 1 tables, or something else. > [INFO ] [CompactionExecutor:77758] cluster_id=87 ip_address=127.1.1.1 > CompactionTask.java:241 - Compacted (8a1cffe0-abb5-11ed-b3fc-8d2ac2c52f0d) 1 > sstables to [...] to level=0. 86.997GiB to 86.997GiB (~99% of original) in > 1,508,155ms. Read Throughput = 59.069MiB/s, Write Throughput = 59.069MiB/s, > Row Throughput = ~20,283/s. 21,375 total partitions merged to 21,375. > Partition merge counts were \{1:21375, } > [ERROR] [CompactionExecutor:4] cluster_id=87 ip_address=127.1.1.1 > CassandraDaemon.java:581 - Exception in thread > Thread[CompactionExecutor:4,1,main] > java.lang.RuntimeException: Not enough space for compaction, estimated > sstables = 1, expected write size = 161228934093 > at > org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100) > at > org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613) > at > org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:834) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18260) Add details to Error message: Not enough space for compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-18260: --- Description: When compaction fails, the log message should list a) the free space available on disk at that point in time and b) perhaps the number and/or size of the source sstables being compacted. Free space can change from one moment to the next, so when the below compaction failed because it needed 161GB, upon checking the server a few minutes later, it had 184GB free. Similarly, the error message mentions it was writing out one SSTable on this STCS table, but its not clear if it was combining X -> 1 tables, or something else. {noformat} [INFO ] [CompactionExecutor:77758] cluster_id=87 ip_address=127.1.1.1 CompactionTask.java:241 - Compacted (8a1cffe0-abb5-11ed-b3fc-8d2ac2c52f0d) 1 sstables to [...] to level=0. 86.997GiB to 86.997GiB (~99% of original) in 1,508,155ms. Read Throughput = 59.069MiB/s, Write Throughput = 59.069MiB/s, Row Throughput = ~20,283/s. 21,375 total partitions merged to 21,375. Partition merge counts were \{1:21375, } [ERROR] [CompactionExecutor:4] cluster_id=87 ip_address=127.1.1.1 CassandraDaemon.java:581 - Exception in thread Thread[CompactionExecutor:4,1,main] java.lang.RuntimeException: Not enough space for compaction, estimated sstables = 1, expected write size = 161228934093 at org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100) at org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613) at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:834) {noformat} was: When compaction fails, the log message should list a) the free space available on disk at that point in time and b) perhaps the number and/or size of the source sstables being compacted. Free space can change from one moment to the next, so when the below compaction failed because it needed 161GB, upon checking the server a few minutes later, it had 184GB free. Similarly, the error message mentions it was writing out one SSTable on this STCS table, but its not clear if it was combining X -> 1 tables, or something else. [INFO ] [CompactionExecutor:77758] cluster_id=87 ip_address=127.1.1.1 CompactionTask.java:241 - Compacted (8a1cffe0-abb5-11ed-b3fc-8d2ac2c52f0d) 1 sstables to [...] to level=0. 86.997GiB to 86.997GiB (~99% of original) in 1,508,155ms. Read Throughput = 59.069MiB/s, Write Throughput = 59.069MiB/s, Row Throughput = ~20,283/s. 21,375 total partitions merged to 21,375. Partition merge counts were \{1:21375, } [ERROR] [CompactionExecutor:4] cluster_id=87 ip_address=127.1.1.1 CassandraDaemon.java:581 - Exception in thread Thread[CompactionExecutor:4,1,main] java.lang.RuntimeException: Not enough space for compaction, estimated sstables = 1, expected write size = 161228934093 at org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100) at org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613) at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:834) > Add detail
[jira] [Updated] (CASSANDRA-18260) Add details to Error message: Not enough space for compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henrik Ingo updated CASSANDRA-18260: Test and Documentation Plan: Tested on laptop with ant testclasslist I'll get myself a circleci account next to run the full testsuite Status: Patch Available (was: In Progress) > Add details to Error message: Not enough space for compaction > -- > > Key: CASSANDRA-18260 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18260 > Project: Cassandra > Issue Type: Improvement > Components: Observability/Logging >Reporter: Brad Schoening >Assignee: Henrik Ingo >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > When compaction fails, the log message should list a) the free space > available on disk at that point in time and b) perhaps the number and/or size > of the source sstables being compacted. > Free space can change from one moment to the next, so when the below > compaction failed because it needed 161GB, upon checking the server a few > minutes later, it had 184GB free. Similarly, the error message mentions it > was writing out one SSTable on this STCS table, but its not clear if it was > combining X -> 1 tables, or something else. > [INFO ] [CompactionExecutor:77758] cluster_id=87 ip_address=127.1.1.1 > CompactionTask.java:241 - Compacted (8a1cffe0-abb5-11ed-b3fc-8d2ac2c52f0d) 1 > sstables to [...] to level=0. 86.997GiB to 86.997GiB (~99% of original) in > 1,508,155ms. Read Throughput = 59.069MiB/s, Write Throughput = 59.069MiB/s, > Row Throughput = ~20,283/s. 21,375 total partitions merged to 21,375. > Partition merge counts were \{1:21375, } > [ERROR] [CompactionExecutor:4] cluster_id=87 ip_address=127.1.1.1 > CassandraDaemon.java:581 - Exception in thread > Thread[CompactionExecutor:4,1,main] > java.lang.RuntimeException: Not enough space for compaction, estimated > sstables = 1, expected write size = 161228934093 > at > org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100) > at > org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613) > at > org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:834) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18260) Add details to Error message: Not enough space for compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-18260: - Change Category: Operability Complexity: Low Hanging Fruit Fix Version/s: 4.0.x 4.1.x 4.x Status: Open (was: Triage Needed) > Add details to Error message: Not enough space for compaction > -- > > Key: CASSANDRA-18260 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18260 > Project: Cassandra > Issue Type: Improvement > Components: Observability/Logging >Reporter: Brad Schoening >Priority: Normal > Fix For: 4.0.x, 4.1.x, 4.x > > > When compaction fails, the log message should list a) the free space > available on disk at that point in time and b) perhaps the number and/or size > of the source sstables being compacted. > Free space can change from one moment to the next, so when the below > compaction failed because it needed 161GB, upon checking the server a few > minutes later, it had 184GB free. Similarly, the error message mentions it > was writing out one SSTable on this STCS table, but its not clear if it was > combining X -> 1 tables, or something else. > [INFO ] [CompactionExecutor:77758] cluster_id=87 ip_address=127.1.1.1 > CompactionTask.java:241 - Compacted (8a1cffe0-abb5-11ed-b3fc-8d2ac2c52f0d) 1 > sstables to [...] to level=0. 86.997GiB to 86.997GiB (~99% of original) in > 1,508,155ms. Read Throughput = 59.069MiB/s, Write Throughput = 59.069MiB/s, > Row Throughput = ~20,283/s. 21,375 total partitions merged to 21,375. > Partition merge counts were \{1:21375, } > [ERROR] [CompactionExecutor:4] cluster_id=87 ip_address=127.1.1.1 > CassandraDaemon.java:581 - Exception in thread > Thread[CompactionExecutor:4,1,main] > java.lang.RuntimeException: Not enough space for compaction, estimated > sstables = 1, expected write size = 161228934093 > at > org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100) > at > org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613) > at > org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:834) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18260) Add details to Error message: Not enough space for compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brad Schoening updated CASSANDRA-18260: --- Description: When compaction fails, the log message should list a) the free space available on disk at that point in time and b) perhaps the number and/or size of the source sstables being compacted. Free space can change from one moment to the next, so when the below compaction failed because it needed 161GB, upon checking the server a few minutes later, it had 184GB free. Similarly, the error message mentions it was writing out one SSTable on this STCS table, but its not clear if it was combining X -> 1 tables, or something else. [INFO ] [CompactionExecutor:77758] cluster_id=87 ip_address=127.1.1.1 CompactionTask.java:241 - Compacted (8a1cffe0-abb5-11ed-b3fc-8d2ac2c52f0d) 1 sstables to [...] to level=0. 86.997GiB to 86.997GiB (~99% of original) in 1,508,155ms. Read Throughput = 59.069MiB/s, Write Throughput = 59.069MiB/s, Row Throughput = ~20,283/s. 21,375 total partitions merged to 21,375. Partition merge counts were \{1:21375, } [ERROR] [CompactionExecutor:4] cluster_id=87 ip_address=127.1.1.1 CassandraDaemon.java:581 - Exception in thread Thread[CompactionExecutor:4,1,main] java.lang.RuntimeException: Not enough space for compaction, estimated sstables = 1, expected write size = 161228934093 at org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100) at org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613) at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:834) was: When compaction fails, the log message should list a) the free space available on disk at that point in time and b) perhaps the number and/or size of the source sstables being compacted. Free space can change from one moment to the next, so when the below compaction failed because it needed 161GB, upon checking the server a few minutes later, it had 184GB free. Similarly, the error message mentions it was writing out one SSTable on this STCS table, but its not clear if it was combining X -> 1 tables, or something else. [INFO ] [CompactionExecutor:77758] cluster_id=87 ip_address=127.1.1.1 CompactionTask.java:241 - Compacted (8a1cffe0-abb5-11ed-b3fc-8d2ac2c52f0d) 1 sstables to [...] to level=0. 86.997GiB to 86.997GiB (~99% of original) in 1,508,155ms. Read Throughput = 59.069MiB/s, Write Throughput = 59.069MiB/s, Row Throughput = ~20,283/s. 21,375 total partitions merged to 21,375. Partition merge counts were \{1:21375, } [ERROR] [CompactionExecutor:4] cluster_id=87 ip_address=127.1.1.1 CassandraDaemon.java:581 - Exception in thread Thread[CompactionExecutor:4,1,main] java.lang.RuntimeException: Not enough space for compaction, estimated sstables = 1, expected write size = 161228934093 at org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100) at org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613) at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:834) > Add details to Error message: Not eno
[jira] [Updated] (CASSANDRA-18260) Add details to Error message: Not enough space for compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brad Schoening updated CASSANDRA-18260: --- Description: When compaction fails, the log message should list a) the free space available on disk at that point in time and b) perhaps the number and/or size of the source sstables being compacted. Free space can change from one moment to the next, so when the below compaction failed because it needed 161GB, upon checking the server a few minutes later, it had 184GB free. Similarly, the error message mentions it was writing out one SSTable on this STCS table, but its not clear if it was combining X -> 1 tables, or something else. [INFO ] [CompactionExecutor:77758] cluster_id=87 ip_address=127.1.1.1 CompactionTask.java:241 - Compacted (8a1cffe0-abb5-11ed-b3fc-8d2ac2c52f0d) 1 sstables to [...] to level=0. 86.997GiB to 86.997GiB (~99% of original) in 1,508,155ms. Read Throughput = 59.069MiB/s, Write Throughput = 59.069MiB/s, Row Throughput = ~20,283/s. 21,375 total partitions merged to 21,375. Partition merge counts were \{1:21375, } [ERROR] [CompactionExecutor:4] cluster_id=87 ip_address=127.1.1.1 CassandraDaemon.java:581 - Exception in thread Thread[CompactionExecutor:4,1,main] java.lang.RuntimeException: Not enough space for compaction, estimated sstables = 1, expected write size = 161228934093 at org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100) at org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613) at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:834) was: When compaction fails, the log message should list a) the free space available on disk at that point in time and b) perhaps the number and/or size of the source sstables being compacted. Free space can change from one moment to the next, so when the below compaction failed because it needed 161GB, upon checking the server a few minutes later, it had 184GB free. Similarly, the error message mentions it was writing out one SSTable on this STCS table, but its not clear if it was combining X -> 1 tables, or something else. [ERROR] [CompactionExecutor:4] cluster_id=87 ip_address=127.1.1.1 CassandraDaemon.java:581 - Exception in thread Thread[CompactionExecutor:4,1,main] java.lang.RuntimeException: Not enough space for compaction, estimated sstables = 1, expected write size = 161228934093 at org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100) at org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613) at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:834) > Add details to Error message: Not enough space for compaction > -- > > Key: CASSANDRA-18260 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18260 > Project: Cassandra > Issue Type: Improvement > Components: Observability/Logging >Reporter: Brad Schoening >Priority: Normal > > When compaction fails, th