How to do cassandra routine maintenance

I am using the cluster with 3 cassandra nodes, the cluster version is 3.0.9. Each day about 200~300 million records are inserted into the cluster. As time goes by, more and more data occupied more and more disk space. Currently,the data distribution on each node is as the following: UN

Re: old big tombstone data file occupy much disk space

After I run nodetool compactionstats -H, it says that: pending tasks: 6 - gps.gpsfullwithstate: 6 id compaction type keyspace table completed total unit progress 56ebd730-8ede-11e7-9754-c981af5d39a9 Validation gps gpsfullwithstate

Re: Cassandra 3.11 is compacting forever

try to do a rolling restart for the cluster before doing a compation On Fri, Sep 1, 2017 at 3:09 PM, Igor Leão wrote: > Some generic errors: > > *[aladdin@ip-172-16-1-10 cassandra]$ tail cassandra.log | grep -i error* > *[aladdin@ip-172-16-1-10 cassandra]$ tail cassandra.log | grep -i excep* > *

Re: Cassandra 3.11 is compacting forever

Some generic errors: *[aladdin@ip-172-16-1-10 cassandra]$ tail cassandra.log | grep -i error* *[aladdin@ip-172-16-1-10 cassandra]$ tail cassandra.log | grep -i excep* *[aladdin@ip-172-16-1-10 cassandra]$ tail cassandra.log | grep -i fail* *[aladdin@ip-172-16-1-10 cassandra]$ tail debug.log | grep

Re: Cassandra snapshot restore with VNODES missing some data

yes looks like I am missing that. Let me test on one node and try a full cluster restore. will update here once I complete my test On Fri, Sep 1, 2017 at 5:01 AM, kurt greaves wrote: > is num_tokens also set to 256? >

Re: How to check if repair is actually successful

At the end of the repair, you should see something like: [2017-09-01 06:59:04,699] Repair completed successfully [2017-09-01 06:59:04,704] Repair command #1 finished in X hour X minutes X seconds On Fri, Sep 1, 2017 at 9:51 AM, Blake Eggleston wrote: > If nodetool repair doesn't return an error

Re: How to check if repair is actually successful

If nodetool repair doesn't return an error, and doesn't hang, the repair completed successfully. On September 1, 2017 at 5:50:53 AM, Akshit Jain (akshit13...@iiitd.ac.in) wrote: Hi, I am performing repair on cassandra cluster. After getting repair status as successful, How to figure out if it is

Re: Cassandra CF Level Metrics (Read, Write Count and Latency)

To be future compatible should consider using `type=Table` instead of `type=ColumnFamily` depending on your version. > not matching with the total read requests the table level metrics for Read/Write latencies will not match the number of requests you've made. This metric is the amount of time it

How to check if repair is actually successful

Hi, I am performing repair on cassandra cluster. After getting repair status as successful, How to figure out if it is successful actually? Is there any way to test it?

Test repair command

​Hi everyone, I'm new to cassandra.I was checking the nodetool repair command. I ran the command and got success but I am not able to figure out how to check repair has actually happened or not? It would be a great help if somebody can suggest a way to do that in terms of data check etc.

Re: old big tombstone data file occupy much disk space

Hi, Well, the command you are using works for me on 3.0.9, I do not have any logs in INFO level when I force a compaction and everything works fine for me. Are you sure there is nothing happening behind the scene ? What dies 'nodetool compactionstats -H' says ? On 1 September 2017 at 12:05, qf z

Re: Cassandra 3.11 is compacting forever

are you seeing any errors in the logs? Is that one compaction still getting stuck?

Re: Cassandra snapshot restore with VNODES missing some data

is num_tokens also set to 256?

Re: old big tombstone data file occupy much disk space

When I trigger the compaction with the full path,  I found nothing in the system.log.  Nothing happens in the  terminal and it just stops there.#calling operation forceUserDefinedCompaction of mbean org.apache.cassandra.db:type=CompactionManager在 2017年9月1日，下午5:06，qf zhou 写道：I

Re: old big tombstone data file occupy much disk space

Whoops sorry I mislead you with cassandra 2.1 behavior, you were right giving your sstable full path , what kind of log do you have when you trigger the compaction with the full path ? On 1 September 2017 at 11:30, Nicolas Guyomar wrote: > Well, not sure why you reached a memory usage limit, but

Re: old big tombstone data file occupy much disk space

Well, not sure why you reached a memory usage limit, but according to the 3.0 branche's code : https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L632 you just need to give the sstable filename, and Cassandra manage to find it b

Re: old big tombstone data file occupy much disk space

I found the following log. What does it mean ? INFO [CompactionExecutor:11] 2017-09-01 16:55:47,909 NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB WARN [RMI TCP Connection(1714)-127.0.0.1] 2017-09-01 17:02:42,516 CompactionManager.java:70

Re: old big tombstone data file occupy much disk space

You should have a log coming from the CompactionManager (in cassandra system.log) when you try the command, what does it says ? On 1 September 2017 at 10:07, qf zhou wrote: > When I run the command, the following occurs and it returns null. > > Is it normal ? > > echo "run -b org.apache.cassa

Re: old big tombstone data file occupy much disk space

When I run the command, the following occurs and it returns null. Is it normal ? echo "run -b org.apache.cassandra.db:type=CompactionManager forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar -l localhost:7199 Welcome to JMX

Re: old big tombstone data file occupy much disk space

Hi, Last time I used forceUserDefinedCompaction, I got myself a headache because I was trying to use a full path like you're doing, but in fact it just need the sstable as parameter Can you just try : echo "run -b org.apache.cassandra.db:type=CompactionManager forceUserDefinedCompaction mc-10096