[jira] [Created] (CASSANDRA-12732) sstableofflinerelevel missing in tools deb build
Zach Aller created CASSANDRA-12732: -- Summary: sstableofflinerelevel missing in tools deb build Key: CASSANDRA-12732 URL: https://issues.apache.org/jira/browse/CASSANDRA-12732 Project: Cassandra Issue Type: Bug Reporter: Zach Aller Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12732) sstableofflinerelevel missing in tools deb package
[ https://issues.apache.org/jira/browse/CASSANDRA-12732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach Aller updated CASSANDRA-12732: --- Summary: sstableofflinerelevel missing in tools deb package (was: sstableofflinerelevel missing in tools deb build) > sstableofflinerelevel missing in tools deb package > -- > > Key: CASSANDRA-12732 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12732 > Project: Cassandra > Issue Type: Bug >Reporter: Zach Aller >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7042) Disk space growth until restart
[
https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745796#comment-14745796
]
Zach Aller commented on CASSANDRA-7042:
---
We are not experiencing the issue anymore it was fixed on some version in the
2.0.x line and we are now on 2.1.9 so it fixed in both versions.
> Disk space growth until restart
> ---
>
> Key: CASSANDRA-7042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7042
> Project: Cassandra
> Issue Type: Bug
> Environment: Ubuntu 12.04
> Sun Java 7
> Cassandra 2.0.6
>Reporter: Zach Aller
> Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot
> 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png,
> after.log, before.log, system.log.9.gz, tabledump_after_restart.txt,
> tabledump_before_restart.txt
>
>
> Cassandra will constantly eat disk space not sure whats causing it the only
> thing that seems to fix it is a restart of cassandra this happens about every
> 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we
> restart cassandra it usually all clears itself up and disks return to normal
> for a while then something triggers its and starts climbing again. Sometimes
> when we restart compactions pending skyrocket and if we restart a second time
> the compactions pending drop off back to a normal level. One other thing to
> note is the space is not free'd until cassandra starts back up and not when
> shutdown.
> I will get a clean log of before and after restarting next time it happens
> and post it.
> Here is a common ERROR in our logs that might be related
> {noformat}
> ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java
> (line 196) Exception in thread Thread[CompactionExecutor:46,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
> at
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(Unknown Source)
> at
> org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58)
> at
> org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
> ... 17 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-7042) Disk space growth until restart
[
https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745796#comment-14745796
]
Zach Aller edited comment on CASSANDRA-7042 at 9/15/15 5:40 PM:
We are not experiencing the issue anymore it was fixed on some version in the
2.0.x line and we are now on 2.1.9 so it fixed in both versions. If my memory
serves me right i could be way wrong i think it was the above patches that
fixed it also i think i remember building cassandra with those and it worked.
was (Author: zaller):
We are not experiencing the issue anymore it was fixed on some version in the
2.0.x line and we are now on 2.1.9 so it fixed in both versions.
> Disk space growth until restart
> ---
>
> Key: CASSANDRA-7042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7042
> Project: Cassandra
> Issue Type: Bug
> Environment: Ubuntu 12.04
> Sun Java 7
> Cassandra 2.0.6
>Reporter: Zach Aller
> Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot
> 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png,
> after.log, before.log, system.log.9.gz, tabledump_after_restart.txt,
> tabledump_before_restart.txt
>
>
> Cassandra will constantly eat disk space not sure whats causing it the only
> thing that seems to fix it is a restart of cassandra this happens about every
> 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we
> restart cassandra it usually all clears itself up and disks return to normal
> for a while then something triggers its and starts climbing again. Sometimes
> when we restart compactions pending skyrocket and if we restart a second time
> the compactions pending drop off back to a normal level. One other thing to
> note is the space is not free'd until cassandra starts back up and not when
> shutdown.
> I will get a clean log of before and after restarting next time it happens
> and post it.
> Here is a common ERROR in our logs that might be related
> {noformat}
> ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java
> (line 196) Exception in thread Thread[CompactionExecutor:46,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
> at
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(Unknown Source)
> at
> org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58)
> at
> org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
> ... 17 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (CASSANDRA-7042) Disk space growth until restart
[
https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zach Aller updated CASSANDRA-7042:
--
Comment: was deleted
(was: Sorry for the late reply, I finally found time to get a build with this
patch in it but we are runing DSE and when i try to start apache cassandra i
get
ERROR 22:28:38,464 Exception encountered during startup
java.lang.RuntimeException:
org.apache.cassandra.exceptions.ConfigurationException: Unable to find
replication strategy class 'org.apache.cassandra.locator.EverywhereStrategy'
The only thing i have found on it is
(http://qnalist.com/questions/4623138/upgrading-cassandra-from-1-2-11-to-2-0))
> Disk space growth until restart
> ---
>
> Key: CASSANDRA-7042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7042
> Project: Cassandra
> Issue Type: Bug
> Environment: Ubuntu 12.04
> Sun Java 7
> Cassandra 2.0.6
>Reporter: Zach Aller
> Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot
> 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png,
> after.log, before.log, system.log.9.gz, tabledump_after_restart.txt,
> tabledump_before_restart.txt
>
>
> Cassandra will constantly eat disk space not sure whats causing it the only
> thing that seems to fix it is a restart of cassandra this happens about every
> 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we
> restart cassandra it usually all clears itself up and disks return to normal
> for a while then something triggers its and starts climbing again. Sometimes
> when we restart compactions pending skyrocket and if we restart a second time
> the compactions pending drop off back to a normal level. One other thing to
> note is the space is not free'd until cassandra starts back up and not when
> shutdown.
> I will get a clean log of before and after restarting next time it happens
> and post it.
> Here is a common ERROR in our logs that might be related
> {noformat}
> ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java
> (line 196) Exception in thread Thread[CompactionExecutor:46,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
> at
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(Unknown Source)
> at
> org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58)
> at
> org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
> ... 17 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7042) Disk space growth until restart
[
https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14120919#comment-14120919
]
Zach Aller commented on CASSANDRA-7042:
---
Sorry for the late reply, I finally found time to get a build with this patch
in it but we are runing DSE and when i try to start apache cassandra i get
ERROR 22:28:38,464 Exception encountered during startup
java.lang.RuntimeException:
org.apache.cassandra.exceptions.ConfigurationException: Unable to find
replication strategy class 'org.apache.cassandra.locator.EverywhereStrategy'
The only thing i have found on it is
(http://qnalist.com/questions/4623138/upgrading-cassandra-from-1-2-11-to-2-0)
> Disk space growth until restart
> ---
>
> Key: CASSANDRA-7042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7042
> Project: Cassandra
> Issue Type: Bug
> Environment: Ubuntu 12.04
> Sun Java 7
> Cassandra 2.0.6
>Reporter: Zach Aller
> Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot
> 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png,
> after.log, before.log, system.log.9.gz, tabledump_after_restart.txt,
> tabledump_before_restart.txt
>
>
> Cassandra will constantly eat disk space not sure whats causing it the only
> thing that seems to fix it is a restart of cassandra this happens about every
> 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we
> restart cassandra it usually all clears itself up and disks return to normal
> for a while then something triggers its and starts climbing again. Sometimes
> when we restart compactions pending skyrocket and if we restart a second time
> the compactions pending drop off back to a normal level. One other thing to
> note is the space is not free'd until cassandra starts back up and not when
> shutdown.
> I will get a clean log of before and after restarting next time it happens
> and post it.
> Here is a common ERROR in our logs that might be related
> {noformat}
> ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java
> (line 196) Exception in thread Thread[CompactionExecutor:46,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
> at
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(Unknown Source)
> at
> org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58)
> at
> org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
> ... 17 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-7042) Disk space growth until restart
[
https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14109138#comment-14109138
]
Zach Aller edited comment on CASSANDRA-7042 at 8/25/14 2:26 PM:
So question about current comapctors is the documentation wrong about it being
set one per core because if that is the case then our box would be set to 4
correct?
concurrent_compactors¶
Sets the number of concurrent compaction processes allowed to run
simultaneously on a node. Defaults to one compaction process per CPU core.
per
(http://www.datastax.com/documentation/cassandra/2.0/cassandra/configuration/configCassandra_yaml_r.html)
was (Author: zaller):
So question about current comapctors is the documentation wrong about it being
set one per core because if that is the case then our box would be set to 4
concurrent_compactors¶
Sets the number of concurrent compaction processes allowed to run
simultaneously on a node. Defaults to one compaction process per CPU core.
per
(http://www.datastax.com/documentation/cassandra/2.0/cassandra/configuration/configCassandra_yaml_r.html)
> Disk space growth until restart
> ---
>
> Key: CASSANDRA-7042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7042
> Project: Cassandra
> Issue Type: Bug
> Environment: Ubuntu 12.04
> Sun Java 7
> Cassandra 2.0.6
>Reporter: Zach Aller
> Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot
> 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png,
> after.log, before.log, system.log.9.gz, tabledump_after_restart.txt,
> tabledump_before_restart.txt
>
>
> Cassandra will constantly eat disk space not sure whats causing it the only
> thing that seems to fix it is a restart of cassandra this happens about every
> 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we
> restart cassandra it usually all clears itself up and disks return to normal
> for a while then something triggers its and starts climbing again. Sometimes
> when we restart compactions pending skyrocket and if we restart a second time
> the compactions pending drop off back to a normal level. One other thing to
> note is the space is not free'd until cassandra starts back up and not when
> shutdown.
> I will get a clean log of before and after restarting next time it happens
> and post it.
> Here is a common ERROR in our logs that might be related
> {noformat}
> ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java
> (line 196) Exception in thread Thread[CompactionExecutor:46,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
> at
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(Unknown Source)
> at
> org.apache.cassandra.io.util.RandomAccessR
[jira] [Comment Edited] (CASSANDRA-7042) Disk space growth until restart
[
https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14109138#comment-14109138
]
Zach Aller edited comment on CASSANDRA-7042 at 8/25/14 2:25 PM:
So question about current comapctors is the documentation wrong about it being
set one per core because if that is the case then our box would be set to 4
concurrent_compactors¶
Sets the number of concurrent compaction processes allowed to run
simultaneously on a node. Defaults to one compaction process per CPU core.
per
(http://www.datastax.com/documentation/cassandra/2.0/cassandra/configuration/configCassandra_yaml_r.html)
was (Author: zaller):
So question about current comapctors is the documentation wrong about it being
set one per core
concurrent_compactors¶
Sets the number of concurrent compaction processes allowed to run
simultaneously on a node. Defaults to one compaction process per CPU core.
per
(http://www.datastax.com/documentation/cassandra/2.0/cassandra/configuration/configCassandra_yaml_r.html)
> Disk space growth until restart
> ---
>
> Key: CASSANDRA-7042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7042
> Project: Cassandra
> Issue Type: Bug
> Environment: Ubuntu 12.04
> Sun Java 7
> Cassandra 2.0.6
>Reporter: Zach Aller
> Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot
> 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png,
> after.log, before.log, system.log.9.gz, tabledump_after_restart.txt,
> tabledump_before_restart.txt
>
>
> Cassandra will constantly eat disk space not sure whats causing it the only
> thing that seems to fix it is a restart of cassandra this happens about every
> 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we
> restart cassandra it usually all clears itself up and disks return to normal
> for a while then something triggers its and starts climbing again. Sometimes
> when we restart compactions pending skyrocket and if we restart a second time
> the compactions pending drop off back to a normal level. One other thing to
> note is the space is not free'd until cassandra starts back up and not when
> shutdown.
> I will get a clean log of before and after restarting next time it happens
> and post it.
> Here is a common ERROR in our logs that might be related
> {noformat}
> ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java
> (line 196) Exception in thread Thread[CompactionExecutor:46,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
> at
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(Unknown Source)
> at
> org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58)
> at
> org.apache.cassand
[jira] [Commented] (CASSANDRA-7042) Disk space growth until restart
[
https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14109138#comment-14109138
]
Zach Aller commented on CASSANDRA-7042:
---
So question about current comapctors is the documentation wrong about it being
set one per core
concurrent_compactors¶
Sets the number of concurrent compaction processes allowed to run
simultaneously on a node. Defaults to one compaction process per CPU core.
per
(http://www.datastax.com/documentation/cassandra/2.0/cassandra/configuration/configCassandra_yaml_r.html)
> Disk space growth until restart
> ---
>
> Key: CASSANDRA-7042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7042
> Project: Cassandra
> Issue Type: Bug
> Environment: Ubuntu 12.04
> Sun Java 7
> Cassandra 2.0.6
>Reporter: Zach Aller
> Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot
> 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png,
> after.log, before.log, system.log.9.gz, tabledump_after_restart.txt,
> tabledump_before_restart.txt
>
>
> Cassandra will constantly eat disk space not sure whats causing it the only
> thing that seems to fix it is a restart of cassandra this happens about every
> 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we
> restart cassandra it usually all clears itself up and disks return to normal
> for a while then something triggers its and starts climbing again. Sometimes
> when we restart compactions pending skyrocket and if we restart a second time
> the compactions pending drop off back to a normal level. One other thing to
> note is the space is not free'd until cassandra starts back up and not when
> shutdown.
> I will get a clean log of before and after restarting next time it happens
> and post it.
> Here is a common ERROR in our logs that might be related
> {noformat}
> ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java
> (line 196) Exception in thread Thread[CompactionExecutor:46,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
> at
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(Unknown Source)
> at
> org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58)
> at
> org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
> ... 17 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (CASSANDRA-7042) Disk space growth until restart
[
https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zach Aller updated CASSANDRA-7042:
--
Attachment: system.log.9.gz
Debug log file
> Disk space growth until restart
> ---
>
> Key: CASSANDRA-7042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7042
> Project: Cassandra
> Issue Type: Bug
> Environment: Ubuntu 12.04
> Sun Java 7
> Cassandra 2.0.6
>Reporter: Zach Aller
> Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot
> 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png,
> after.log, before.log, system.log.9.gz, tabledump_after_restart.txt,
> tabledump_before_restart.txt
>
>
> Cassandra will constantly eat disk space not sure whats causing it the only
> thing that seems to fix it is a restart of cassandra this happens about every
> 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we
> restart cassandra it usually all clears itself up and disks return to normal
> for a while then something triggers its and starts climbing again. Sometimes
> when we restart compactions pending skyrocket and if we restart a second time
> the compactions pending drop off back to a normal level. One other thing to
> note is the space is not free'd until cassandra starts back up and not when
> shutdown.
> I will get a clean log of before and after restarting next time it happens
> and post it.
> Here is a common ERROR in our logs that might be related
> {noformat}
> ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java
> (line 196) Exception in thread Thread[CompactionExecutor:46,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
> at
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(Unknown Source)
> at
> org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58)
> at
> org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
> ... 17 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7042) Disk space growth until restart
[
https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14108724#comment-14108724
]
Zach Aller commented on CASSANDRA-7042:
---
I uploaded the log now i will lower concurrent compactors to see if it helps
and report back after its run for a while
> Disk space growth until restart
> ---
>
> Key: CASSANDRA-7042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7042
> Project: Cassandra
> Issue Type: Bug
> Environment: Ubuntu 12.04
> Sun Java 7
> Cassandra 2.0.6
>Reporter: Zach Aller
> Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot
> 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png,
> after.log, before.log, system.log.9.gz, tabledump_after_restart.txt,
> tabledump_before_restart.txt
>
>
> Cassandra will constantly eat disk space not sure whats causing it the only
> thing that seems to fix it is a restart of cassandra this happens about every
> 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we
> restart cassandra it usually all clears itself up and disks return to normal
> for a while then something triggers its and starts climbing again. Sometimes
> when we restart compactions pending skyrocket and if we restart a second time
> the compactions pending drop off back to a normal level. One other thing to
> note is the space is not free'd until cassandra starts back up and not when
> shutdown.
> I will get a clean log of before and after restarting next time it happens
> and post it.
> Here is a common ERROR in our logs that might be related
> {noformat}
> ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java
> (line 196) Exception in thread Thread[CompactionExecutor:46,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
> at
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(Unknown Source)
> at
> org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58)
> at
> org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
> ... 17 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Issue Comment Deleted] (CASSANDRA-7042) Disk space growth until restart
[
https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zach Aller updated CASSANDRA-7042:
--
Comment: was deleted
(was: Debug log file)
> Disk space growth until restart
> ---
>
> Key: CASSANDRA-7042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7042
> Project: Cassandra
> Issue Type: Bug
> Environment: Ubuntu 12.04
> Sun Java 7
> Cassandra 2.0.6
>Reporter: Zach Aller
> Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot
> 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png,
> after.log, before.log, system.log.9.gz, tabledump_after_restart.txt,
> tabledump_before_restart.txt
>
>
> Cassandra will constantly eat disk space not sure whats causing it the only
> thing that seems to fix it is a restart of cassandra this happens about every
> 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we
> restart cassandra it usually all clears itself up and disks return to normal
> for a while then something triggers its and starts climbing again. Sometimes
> when we restart compactions pending skyrocket and if we restart a second time
> the compactions pending drop off back to a normal level. One other thing to
> note is the space is not free'd until cassandra starts back up and not when
> shutdown.
> I will get a clean log of before and after restarting next time it happens
> and post it.
> Here is a common ERROR in our logs that might be related
> {noformat}
> ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java
> (line 196) Exception in thread Thread[CompactionExecutor:46,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
> at
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(Unknown Source)
> at
> org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58)
> at
> org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
> ... 17 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7042) Disk space growth until restart
[
https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14106866#comment-14106866
]
Zach Aller commented on CASSANDRA-7042:
---
Ok, I will still probably try to grab a debug log then after that try lowering
concurrent compactors to see if it helps the issue.
> Disk space growth until restart
> ---
>
> Key: CASSANDRA-7042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7042
> Project: Cassandra
> Issue Type: Bug
> Environment: Ubuntu 12.04
> Sun Java 7
> Cassandra 2.0.6
>Reporter: Zach Aller
> Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot
> 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png,
> after.log, before.log, tabledump_after_restart.txt,
> tabledump_before_restart.txt
>
>
> Cassandra will constantly eat disk space not sure whats causing it the only
> thing that seems to fix it is a restart of cassandra this happens about every
> 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we
> restart cassandra it usually all clears itself up and disks return to normal
> for a while then something triggers its and starts climbing again. Sometimes
> when we restart compactions pending skyrocket and if we restart a second time
> the compactions pending drop off back to a normal level. One other thing to
> note is the space is not free'd until cassandra starts back up and not when
> shutdown.
> I will get a clean log of before and after restarting next time it happens
> and post it.
> Here is a common ERROR in our logs that might be related
> {noformat}
> ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java
> (line 196) Exception in thread Thread[CompactionExecutor:46,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
> at
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(Unknown Source)
> at
> org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58)
> at
> org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
> ... 17 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7042) Disk space growth until restart
[
https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14106842#comment-14106842
]
Zach Aller commented on CASSANDRA-7042:
---
We are using amazon ec2 i2.xlarge so one 800GB SSD
> Disk space growth until restart
> ---
>
> Key: CASSANDRA-7042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7042
> Project: Cassandra
> Issue Type: Bug
> Environment: Ubuntu 12.04
> Sun Java 7
> Cassandra 2.0.6
>Reporter: Zach Aller
> Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot
> 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png,
> after.log, before.log, tabledump_after_restart.txt,
> tabledump_before_restart.txt
>
>
> Cassandra will constantly eat disk space not sure whats causing it the only
> thing that seems to fix it is a restart of cassandra this happens about every
> 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we
> restart cassandra it usually all clears itself up and disks return to normal
> for a while then something triggers its and starts climbing again. Sometimes
> when we restart compactions pending skyrocket and if we restart a second time
> the compactions pending drop off back to a normal level. One other thing to
> note is the space is not free'd until cassandra starts back up and not when
> shutdown.
> I will get a clean log of before and after restarting next time it happens
> and post it.
> Here is a common ERROR in our logs that might be related
> {noformat}
> ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java
> (line 196) Exception in thread Thread[CompactionExecutor:46,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
> at
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(Unknown Source)
> at
> org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58)
> at
> org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
> ... 17 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7042) Disk space growth until restart
[
https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14106807#comment-14106807
]
Zach Aller commented on CASSANDRA-7042:
---
Our compaction limit is set to 0 and compaction memory 768MB. Concurrent
compactors is set to the default and multithreading is off and compaction pre
heat cache is also off. I am going to be getting a debug log today hopefully
with this (log4j.logger.org.apache.cassandra.db.compaction=DEBUG) so we will
see what the reveals.
> Disk space growth until restart
> ---
>
> Key: CASSANDRA-7042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7042
> Project: Cassandra
> Issue Type: Bug
> Environment: Ubuntu 12.04
> Sun Java 7
> Cassandra 2.0.6
>Reporter: Zach Aller
> Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot
> 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png,
> after.log, before.log, tabledump_after_restart.txt,
> tabledump_before_restart.txt
>
>
> Cassandra will constantly eat disk space not sure whats causing it the only
> thing that seems to fix it is a restart of cassandra this happens about every
> 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we
> restart cassandra it usually all clears itself up and disks return to normal
> for a while then something triggers its and starts climbing again. Sometimes
> when we restart compactions pending skyrocket and if we restart a second time
> the compactions pending drop off back to a normal level. One other thing to
> note is the space is not free'd until cassandra starts back up and not when
> shutdown.
> I will get a clean log of before and after restarting next time it happens
> and post it.
> Here is a common ERROR in our logs that might be related
> {noformat}
> ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java
> (line 196) Exception in thread Thread[CompactionExecutor:46,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
> at
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(Unknown Source)
> at
> org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58)
> at
> org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
> ... 17 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7042) Disk space growth until restart
[
https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105605#comment-14105605
]
Zach Aller commented on CASSANDRA-7042:
---
No, we never do a truncate, and our commit log folder is around 1GB in size.
When we restart the data directory is what shrinks in size.
> Disk space growth until restart
> ---
>
> Key: CASSANDRA-7042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7042
> Project: Cassandra
> Issue Type: Bug
> Environment: Ubuntu 12.04
> Sun Java 7
> Cassandra 2.0.6
>Reporter: Zach Aller
> Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot
> 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png,
> after.log, before.log, tabledump_after_restart.txt,
> tabledump_before_restart.txt
>
>
> Cassandra will constantly eat disk space not sure whats causing it the only
> thing that seems to fix it is a restart of cassandra this happens about every
> 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we
> restart cassandra it usually all clears itself up and disks return to normal
> for a while then something triggers its and starts climbing again. Sometimes
> when we restart compactions pending skyrocket and if we restart a second time
> the compactions pending drop off back to a normal level. One other thing to
> note is the space is not free'd until cassandra starts back up and not when
> shutdown.
> I will get a clean log of before and after restarting next time it happens
> and post it.
> Here is a common ERROR in our logs that might be related
> {noformat}
> ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java
> (line 196) Exception in thread Thread[CompactionExecutor:46,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
> at
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(Unknown Source)
> at
> org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58)
> at
> org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
> ... 17 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7042) Disk space growth until restart
[
https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997960#comment-13997960
]
Zach Aller commented on CASSANDRA-7042:
---
Interesting, we are working on a script to reproduce that dose not require a
bunch of dependancies and our whole infrastructure setup as a side note just
recently deleted a key space and when looking at the data directories there
where a bunch of files left behind on a few nodes that exhibited the issue.
> Disk space growth until restart
> ---
>
> Key: CASSANDRA-7042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7042
> Project: Cassandra
> Issue Type: Bug
> Environment: Ubuntu 12.04
> Sun Java 7
> Cassandra 2.0.6
>Reporter: Zach Aller
> Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot
> 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png,
> after.log, before.log, tabledump_after_restart.txt,
> tabledump_before_restart.txt
>
>
> Cassandra will constantly eat disk space not sure whats causing it the only
> thing that seems to fix it is a restart of cassandra this happens about every
> 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we
> restart cassandra it usually all clears itself up and disks return to normal
> for a while then something triggers its and starts climbing again. Sometimes
> when we restart compactions pending skyrocket and if we restart a second time
> the compactions pending drop off back to a normal level. One other thing to
> note is the space is not free'd until cassandra starts back up and not when
> shutdown.
> I will get a clean log of before and after restarting next time it happens
> and post it.
> Here is a common ERROR in our logs that might be related
> {noformat}
> ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java
> (line 196) Exception in thread Thread[CompactionExecutor:46,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
> at
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(Unknown Source)
> at
> org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58)
> at
> org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
> ... 17 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7042) Disk space growth until restart
[ https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13983482#comment-13983482 ] Zach Aller commented on CASSANDRA-7042: --- Just thought I would also mention setup a brand new 3 node test cluster completely fresh and was able to reproduce. > Disk space growth until restart > --- > > Key: CASSANDRA-7042 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7042 > Project: Cassandra > Issue Type: Bug > Environment: Ubuntu 12.04 > Sun Java 7 > Cassandra 2.0.6 >Reporter: Zach Aller > Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot > 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png, > after.log, before.log, tabledump_after_restart.txt, > tabledump_before_restart.txt > > > Cassandra will constantly eat disk space not sure whats causing it the only > thing that seems to fix it is a restart of cassandra this happens about every > 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we > restart cassandra it usually all clears itself up and disks return to normal > for a while then something triggers its and starts climbing again. Sometimes > when we restart compactions pending skyrocket and if we restart a second time > the compactions pending drop off back to a normal level. One other thing to > note is the space is not free'd until cassandra starts back up and not when > shutdown. > I will get a clean log of before and after restarting next time it happens > and post it. > Here is a common ERROR in our logs that might be related > ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java > (line 196) Exception in thread Thread[CompactionExecutor:46,1,main] > java.lang.RuntimeException: java.io.FileNotFoundException: > /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db > (No such file or directory) > at > org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53) > at > org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355) > at > org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67) > at > org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161) > at > org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194) > at > org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197) > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > at java.util.concurrent.FutureTask.run(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.FileNotFoundException: > /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db > (No such file or directory) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(Unknown Source) > at > org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58) > at > org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35) > at > org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49) > ... 17 more -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7042) Disk space growth until restart
[
https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13980094#comment-13980094
]
Zach Aller commented on CASSANDRA-7042:
---
Here is a schema in question
CREATE TABLE grid (
data_id text,
cylinder text,
value blob,
PRIMARY KEY (data_id, cylinder)
) WITH COMPACT STORAGE AND
bloom_filter_fp_chance=0.10 AND
caching='ALL' AND
comment='' AND
dclocal_read_repair_chance=0.00 AND
gc_grace_seconds=0 AND
index_interval=128 AND
read_repair_chance=0.10 AND
populate_io_cache_on_flush='false' AND
default_time_to_live=0 AND
speculative_retry='99.0PERCENTILE' AND
memtable_flush_period_in_ms=0 AND
compaction={'sstable_size_in_mb': '160', 'class':
'LeveledCompactionStrategy'} AND
compression={};
> Disk space growth until restart
> ---
>
> Key: CASSANDRA-7042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7042
> Project: Cassandra
> Issue Type: Bug
> Environment: Ubuntu 12.04
> Sun Java 7
> Cassandra 2.0.6
>Reporter: Zach Aller
> Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot
> 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png,
> after.log, before.log, tabledump_after_restart.txt,
> tabledump_before_restart.txt
>
>
> Cassandra will constantly eat disk space not sure whats causing it the only
> thing that seems to fix it is a restart of cassandra this happens about every
> 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we
> restart cassandra it usually all clears itself up and disks return to normal
> for a while then something triggers its and starts climbing again. Sometimes
> when we restart compactions pending skyrocket and if we restart a second time
> the compactions pending drop off back to a normal level. One other thing to
> note is the space is not free'd until cassandra starts back up and not when
> shutdown.
> I will get a clean log of before and after restarting next time it happens
> and post it.
> Here is a common ERROR in our logs that might be related
> ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java
> (line 196) Exception in thread Thread[CompactionExecutor:46,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
> at
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(Unknown Source)
> at
> org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58)
> at
> org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
> ... 17 more
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (CASSANDRA-7042) Disk space growth until restart
[ https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach Aller updated CASSANDRA-7042: -- Attachment: tabledump_before_restart.txt tabledump_after_restart.txt This is a dump of the directory for the cf that has the issues as you can see file count increases. It seems like files are not being deleted when they should be. The before file is taken when cassandra was in a stopped state the after is when cassandra had been started up again. > Disk space growth until restart > --- > > Key: CASSANDRA-7042 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7042 > Project: Cassandra > Issue Type: Bug > Environment: Ubuntu 12.04 > Sun Java 7 > Cassandra 2.0.6 >Reporter: Zach Aller > Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot > 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png, > after.log, before.log, tabledump_after_restart.txt, > tabledump_before_restart.txt > > > Cassandra will constantly eat disk space not sure whats causing it the only > thing that seems to fix it is a restart of cassandra this happens about every > 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we > restart cassandra it usually all clears itself up and disks return to normal > for a while then something triggers its and starts climbing again. Sometimes > when we restart compactions pending skyrocket and if we restart a second time > the compactions pending drop off back to a normal level. One other thing to > note is the space is not free'd until cassandra starts back up and not when > shutdown. > I will get a clean log of before and after restarting next time it happens > and post it. > Here is a common ERROR in our logs that might be related > ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java > (line 196) Exception in thread Thread[CompactionExecutor:46,1,main] > java.lang.RuntimeException: java.io.FileNotFoundException: > /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db > (No such file or directory) > at > org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53) > at > org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355) > at > org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67) > at > org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161) > at > org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194) > at > org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197) > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > at java.util.concurrent.FutureTask.run(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.FileNotFoundException: > /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db > (No such file or directory) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(Unknown Source) > at > org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58) > at > org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35) > at > org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49) > ... 17 more -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-7042) Disk space growth until restart
[ https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach Aller updated CASSANDRA-7042: -- Attachment: Screen Shot 2014-04-22 at 1.40.41 PM.png > Disk space growth until restart > --- > > Key: CASSANDRA-7042 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7042 > Project: Cassandra > Issue Type: Bug > Environment: Ubuntu 12.04 > Sun Java 7 > Cassandra 2.0.6 >Reporter: Zach Aller > Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot > 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png, > after.log, before.log > > > Cassandra will constantly eat disk space not sure whats causing it the only > thing that seems to fix it is a restart of cassandra this happens about every > 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we > restart cassandra it usually all clears itself up and disks return to normal > for a while then something triggers its and starts climbing again. Sometimes > when we restart compactions pending skyrocket and if we restart a second time > the compactions pending drop off back to a normal level. One other thing to > note is the space is not free'd until cassandra starts back up and not when > shutdown. > I will get a clean log of before and after restarting next time it happens > and post it. > Here is a common ERROR in our logs that might be related > ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java > (line 196) Exception in thread Thread[CompactionExecutor:46,1,main] > java.lang.RuntimeException: java.io.FileNotFoundException: > /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db > (No such file or directory) > at > org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53) > at > org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355) > at > org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67) > at > org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161) > at > org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194) > at > org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197) > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > at java.util.concurrent.FutureTask.run(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.FileNotFoundException: > /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db > (No such file or directory) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(Unknown Source) > at > org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58) > at > org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35) > at > org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49) > ... 17 more -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (CASSANDRA-7042) Disk space growth until restart
[ https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976386#comment-13976386 ] Zach Aller edited comment on CASSANDRA-7042 at 4/22/14 5:23 AM: Doing nothing they just show up in the logs during normal use our tables are all ttl'd set to about a day with around 30GB or so of data churn. Upgraded to 2.0.7 and still having the issue, here is a error for the 2.0.7 version not even sure if the error is related to the space growth but it is the only bit of info in the logs that might suggest it. Willing to help you guys out anyway I can would love to get this fixed its pretty much our last major issue with keeping our cluster stable. ERROR [CompactionExecutor:112] 2014-04-21 21:26:35,109 CassandraDaemon.java (line 198) Exception in thread Thread[CompactionExecutor:112,1,main] java.lang.RuntimeException: java.io.FileNotFoundException: /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-580812-Data.db (No such file or directory) at org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53) at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355) at org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67) at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161) was (Author: zaller): Doing nothing they just show up in the logs during normal use our tables are all ttl'd set to about a day with around 30GB or so of data churn. Upgraded to 2.0.7 and still having the issue, here is a error for the 2.0.7 version. Willing to help you guys out anyway I can would love to get this fixed its pretty much our last major issue with keeping our cluster stable. ERROR [CompactionExecutor:112] 2014-04-21 21:26:35,109 CassandraDaemon.java (line 198) Exception in thread Thread[CompactionExecutor:112,1,main] java.lang.RuntimeException: java.io.FileNotFoundException: /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-580812-Data.db (No such file or directory) at org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53) at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355) at org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67) at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161) > Disk space growth until restart > --- > > Key: CASSANDRA-7042 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7042 > Project: Cassandra > Issue Type: Bug > Environment: Ubuntu 12.04 > Sun Java 7 > Cassandra 2.0.6 >Reporter: Zach Aller > Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot > 2014-04-18 at 11.47.30 AM.png, after.log, before.log > > > Cassandra will constantly eat disk space not sure whats causing it the only > thing that seems to fix it is a restart of cassandra this happens about every > 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we > restart cassandra it usually all clears itself up and disks return to normal > for a while then something triggers its and starts climbing again. Sometimes > when we restart compactions pending skyrocket and if we restart a second time > the compactions pending drop off back to a normal level. One other thing to > note is the space is not free'd until cassandra starts back up and not when > shutdown. > I will get a clean log of before and after restarting next time it happens > and post it. > Here is a common ERROR in our logs that might be related > ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java > (line 196) Exception in thread Thread[CompactionExecutor:46,1,main] > java.lang.RuntimeException: java.io.FileNotFoundException: > /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db > (No such file or directory) > at > org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53) > at > org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355) > at > org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67) > at > org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161) > at > org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194) > at > org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126) >
[jira] [Comment Edited] (CASSANDRA-7042) Disk space growth until restart
[ https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976392#comment-13976392 ] Zach Aller edited comment on CASSANDRA-7042 at 4/22/14 5:02 AM: Here is some screen shots for datastax opscenter. The dips are related to either running cleanup or changing the gc_grace period down after learning that since we ttl all our data we don't have to run repairs within gc_grace. You can see the spikes in the total disk space they will basicly continue until it runs our of space or we restart cassandra. was (Author: zaller): Here is some screen shots for datastax opscenter. The dips are related to either running cleanup or changing the gc_grace period down after learning that since we ttl all our data we don't have to run repairs within gc_grace. You can see the spikes in the total disk space. > Disk space growth until restart > --- > > Key: CASSANDRA-7042 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7042 > Project: Cassandra > Issue Type: Bug > Environment: Ubuntu 12.04 > Sun Java 7 > Cassandra 2.0.6 >Reporter: Zach Aller > Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot > 2014-04-18 at 11.47.30 AM.png, after.log, before.log > > > Cassandra will constantly eat disk space not sure whats causing it the only > thing that seems to fix it is a restart of cassandra this happens about every > 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we > restart cassandra it usually all clears itself up and disks return to normal > for a while then something triggers its and starts climbing again. Sometimes > when we restart compactions pending skyrocket and if we restart a second time > the compactions pending drop off back to a normal level. One other thing to > note is the space is not free'd until cassandra starts back up and not when > shutdown. > I will get a clean log of before and after restarting next time it happens > and post it. > Here is a common ERROR in our logs that might be related > ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java > (line 196) Exception in thread Thread[CompactionExecutor:46,1,main] > java.lang.RuntimeException: java.io.FileNotFoundException: > /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db > (No such file or directory) > at > org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53) > at > org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355) > at > org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67) > at > org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161) > at > org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194) > at > org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197) > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > at java.util.concurrent.FutureTask.run(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.FileNotFoundException: > /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db > (No such file or directory) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(Unknown Source) > at > org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58) > at > org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35) > at > org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49) > ... 17 more -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (CASSANDRA-7042) Disk space growth until restart
[ https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976392#comment-13976392 ] Zach Aller edited comment on CASSANDRA-7042 at 4/22/14 5:01 AM: Here is some screen shots for datastax opscenter. The dips are related to either running cleanup or changing the gc_grace period down after learning that since we ttl all our data we don't have to run repairs within gc_grace. You can see the spikes in the total disk space. was (Author: zaller): Here is some screen shots for datastax opscenter. The dips are related to either running cleanup or changing the gc_grace period down after learning that since we ttl all our data we don't have to run repairs within gc_grace > Disk space growth until restart > --- > > Key: CASSANDRA-7042 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7042 > Project: Cassandra > Issue Type: Bug > Environment: Ubuntu 12.04 > Sun Java 7 > Cassandra 2.0.6 >Reporter: Zach Aller > Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot > 2014-04-18 at 11.47.30 AM.png, after.log, before.log > > > Cassandra will constantly eat disk space not sure whats causing it the only > thing that seems to fix it is a restart of cassandra this happens about every > 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we > restart cassandra it usually all clears itself up and disks return to normal > for a while then something triggers its and starts climbing again. Sometimes > when we restart compactions pending skyrocket and if we restart a second time > the compactions pending drop off back to a normal level. One other thing to > note is the space is not free'd until cassandra starts back up and not when > shutdown. > I will get a clean log of before and after restarting next time it happens > and post it. > Here is a common ERROR in our logs that might be related > ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java > (line 196) Exception in thread Thread[CompactionExecutor:46,1,main] > java.lang.RuntimeException: java.io.FileNotFoundException: > /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db > (No such file or directory) > at > org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53) > at > org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355) > at > org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67) > at > org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161) > at > org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194) > at > org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197) > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > at java.util.concurrent.FutureTask.run(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.FileNotFoundException: > /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db > (No such file or directory) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(Unknown Source) > at > org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58) > at > org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35) > at > org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49) > ... 17 more -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-7042) Disk space growth until restart
[ https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach Aller updated CASSANDRA-7042: -- Attachment: Screen Shot 2014-04-18 at 11.47.30 AM.png Screen Shot 2014-04-17 at 11.07.24 AM.png Here is some screen shots for datastax opscenter. The dips are related to either running cleanup or changing the gc_grace period down after learning that since we ttl all our data we don't have to run repairs within gc_grace > Disk space growth until restart > --- > > Key: CASSANDRA-7042 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7042 > Project: Cassandra > Issue Type: Bug > Environment: Ubuntu 12.04 > Sun Java 7 > Cassandra 2.0.6 >Reporter: Zach Aller > Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot > 2014-04-18 at 11.47.30 AM.png, after.log, before.log > > > Cassandra will constantly eat disk space not sure whats causing it the only > thing that seems to fix it is a restart of cassandra this happens about every > 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we > restart cassandra it usually all clears itself up and disks return to normal > for a while then something triggers its and starts climbing again. Sometimes > when we restart compactions pending skyrocket and if we restart a second time > the compactions pending drop off back to a normal level. One other thing to > note is the space is not free'd until cassandra starts back up and not when > shutdown. > I will get a clean log of before and after restarting next time it happens > and post it. > Here is a common ERROR in our logs that might be related > ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java > (line 196) Exception in thread Thread[CompactionExecutor:46,1,main] > java.lang.RuntimeException: java.io.FileNotFoundException: > /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db > (No such file or directory) > at > org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53) > at > org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355) > at > org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67) > at > org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161) > at > org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194) > at > org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197) > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > at java.util.concurrent.FutureTask.run(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.FileNotFoundException: > /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db > (No such file or directory) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(Unknown Source) > at > org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58) > at > org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35) > at > org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49) > ... 17 more -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (CASSANDRA-7042) Disk space growth until restart
[ https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976386#comment-13976386 ] Zach Aller edited comment on CASSANDRA-7042 at 4/22/14 4:56 AM: Doing nothing they just show up in the logs during normal use our tables are all ttl'd set to about a day with around 30GB or so of data churn. Upgraded to 2.0.7 and still having the issue, here is a error for the 2.0.7 version. Willing to help you guys out anyway I can would love to get this fixed its pretty much our last major issue with keeping our cluster stable. ERROR [CompactionExecutor:112] 2014-04-21 21:26:35,109 CassandraDaemon.java (line 198) Exception in thread Thread[CompactionExecutor:112,1,main] java.lang.RuntimeException: java.io.FileNotFoundException: /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-580812-Data.db (No such file or directory) at org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53) at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355) at org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67) at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161) was (Author: zaller): Doing nothing they just show up in the logs during normal use our tables are all ttl'd set to about a day with around 30GB or so of data churn. Upgraded to 2.0.7 and still having the issue here is a error for the 2.0.7 version. Willing to help you guys out anyway I can would love to get this fixed its pretty much our last major issue with keeping our cluster stable. ERROR [CompactionExecutor:112] 2014-04-21 21:26:35,109 CassandraDaemon.java (line 198) Exception in thread Thread[CompactionExecutor:112,1,main] java.lang.RuntimeException: java.io.FileNotFoundException: /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-580812-Data.db (No such file or directory) at org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53) at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355) at org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67) at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161) > Disk space growth until restart > --- > > Key: CASSANDRA-7042 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7042 > Project: Cassandra > Issue Type: Bug > Environment: Ubuntu 12.04 > Sun Java 7 > Cassandra 2.0.6 >Reporter: Zach Aller > Attachments: after.log, before.log > > > Cassandra will constantly eat disk space not sure whats causing it the only > thing that seems to fix it is a restart of cassandra this happens about every > 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we > restart cassandra it usually all clears itself up and disks return to normal > for a while then something triggers its and starts climbing again. Sometimes > when we restart compactions pending skyrocket and if we restart a second time > the compactions pending drop off back to a normal level. One other thing to > note is the space is not free'd until cassandra starts back up and not when > shutdown. > I will get a clean log of before and after restarting next time it happens > and post it. > Here is a common ERROR in our logs that might be related > ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java > (line 196) Exception in thread Thread[CompactionExecutor:46,1,main] > java.lang.RuntimeException: java.io.FileNotFoundException: > /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db > (No such file or directory) > at > org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53) > at > org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355) > at > org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67) > at > org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161) > at > org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194) > at > org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassa
[jira] [Commented] (CASSANDRA-7042) Disk space growth until restart
[ https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976386#comment-13976386 ] Zach Aller commented on CASSANDRA-7042: --- Doing nothing they just show up in the logs during normal use our tables are all ttl'd set to about a day with around 30GB or so of data churn. Upgraded to 2.0.7 and still having the issue here is a error for the 2.0.7 version. Willing to help you guys out anyway I can would love to get this fixed its pretty much our last major issue with keeping our cluster stable. ERROR [CompactionExecutor:112] 2014-04-21 21:26:35,109 CassandraDaemon.java (line 198) Exception in thread Thread[CompactionExecutor:112,1,main] java.lang.RuntimeException: java.io.FileNotFoundException: /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-580812-Data.db (No such file or directory) at org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53) at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355) at org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67) at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161) > Disk space growth until restart > --- > > Key: CASSANDRA-7042 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7042 > Project: Cassandra > Issue Type: Bug > Environment: Ubuntu 12.04 > Sun Java 7 > Cassandra 2.0.6 >Reporter: Zach Aller > Attachments: after.log, before.log > > > Cassandra will constantly eat disk space not sure whats causing it the only > thing that seems to fix it is a restart of cassandra this happens about every > 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we > restart cassandra it usually all clears itself up and disks return to normal > for a while then something triggers its and starts climbing again. Sometimes > when we restart compactions pending skyrocket and if we restart a second time > the compactions pending drop off back to a normal level. One other thing to > note is the space is not free'd until cassandra starts back up and not when > shutdown. > I will get a clean log of before and after restarting next time it happens > and post it. > Here is a common ERROR in our logs that might be related > ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java > (line 196) Exception in thread Thread[CompactionExecutor:46,1,main] > java.lang.RuntimeException: java.io.FileNotFoundException: > /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db > (No such file or directory) > at > org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53) > at > org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355) > at > org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67) > at > org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161) > at > org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194) > at > org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197) > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > at java.util.concurrent.FutureTask.run(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.FileNotFoundException: > /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db > (No such file or directory) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(Unknown Source) > at > org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58) > at > org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35) >
[jira] [Updated] (CASSANDRA-7042) Disk space growth until restart
[ https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach Aller updated CASSANDRA-7042: -- Attachment: after.log before.log > Disk space growth until restart > --- > > Key: CASSANDRA-7042 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7042 > Project: Cassandra > Issue Type: Bug > Environment: Ubuntu 12.04 > Sun Java 7 > Cassandra 2.0.6 >Reporter: Zach Aller >Priority: Critical > Attachments: after.log, before.log > > > Cassandra will constantly eat disk space not sure whats causing it the only > thing that seems to fix it is a restart of cassandra this happens about every > 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we > restart cassandra it usually all clears itself up and disks return to normal > for a while then something triggers its and starts climbing again. Sometimes > when we restart compactions pending skyrocket and if we restart a second time > the compactions pending drop off back to a normal level. One other thing to > note is the space is not free'd until cassandra starts back up and not when > shutdown. > I will get a clean log of before and after restarting next time it happens > and post it. > Here is a common ERROR in our logs that might be related > ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java > (line 196) Exception in thread Thread[CompactionExecutor:46,1,main] > java.lang.RuntimeException: java.io.FileNotFoundException: > /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db > (No such file or directory) > at > org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53) > at > org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355) > at > org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67) > at > org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161) > at > org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194) > at > org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197) > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > at java.util.concurrent.FutureTask.run(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.FileNotFoundException: > /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db > (No such file or directory) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(Unknown Source) > at > org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58) > at > org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35) > at > org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49) > ... 17 more -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (CASSANDRA-7042) Disk space growth until restart
Zach Aller created CASSANDRA-7042: - Summary: Disk space growth until restart Key: CASSANDRA-7042 URL: https://issues.apache.org/jira/browse/CASSANDRA-7042 Project: Cassandra Issue Type: Bug Environment: Ubuntu 12.04 Sun Java 7 Cassandra 2.0.6 Reporter: Zach Aller Priority: Critical Cassandra will constantly eat disk space not sure whats causing it the only thing that seems to fix it is a restart of cassandra this happens about every 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we restart cassandra it usually all clears itself up and disks return to normal for a while then something triggers its and starts climbing again. Sometimes when we restart compactions pending skyrocket and if we restart a second time the compactions pending drop off back to a normal level. One other thing to note is the space is not free'd until cassandra starts back up and not when shutdown. I will get a clean log of before and after restarting next time it happens and post it. Here is a common ERROR in our logs that might be related ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java (line 196) Exception in thread Thread[CompactionExecutor:46,1,main] java.lang.RuntimeException: java.io.FileNotFoundException: /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db (No such file or directory) at org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53) at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355) at org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67) at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161) at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194) at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258) at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.io.FileNotFoundException: /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db (No such file or directory) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.(Unknown Source) at org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58) at org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35) at org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49) ... 17 more -- This message was sent by Atlassian JIRA (v6.2#6252)
