Loss of an Entire AZ in a Three-AZ Cassandra Cluster

2019-03-08 Thread Fd Habash 
Assume you have a 30 node cluster distributed across three AZ’s with an RF of 
3. Trying to come up with a runbook to manage multi-nodes failure as a result 
of …

- Loss of an entire AZ1
- Loss of multiple nodes in AZ2
- AZ3 unaffected. No node loss

Is this is most optimal plan. Replacing dead nodes via bootstrapping  …

1. Replace seeds nodes first (via bootstrap)
2. Bootstrap the few nodes in AZ2
3. Bootstrap all nodes in AZ1
4. Run a cluster repair.

Do you wait to bootstrap everything before running repair or do you repair per 
node?
Did I miss anything? 


Thank you

too many logDroppedMessages and StatusLogger

2019-03-08 Thread Marco Gasparini 
Hi all,

I cannot understand why I get the following logs, they appear every day at
not fixed period of time. I saw them every 2 minutes or every 10 seconds, I
cannot find any pattern.
I took this very example here during an heavy workload of writes and reads
but I get them also during a very little workload and without any active
compaction/repair/streaming process and no high cpu/memory/iowait usage.

2019-03-08 01:49:47,868 INFO  [ScheduledTasks:1] MessagingService.java:1246
> logDroppedMessages READ messages were dropped in last 5000 ms: 0 internal
> and 1 cross node. Mean internal dropped latency: 6357 ms and Mean
> cross-node dropped latency: 6556 ms
> 2019-03-08 01:49:47,868 INFO  [ScheduledTasks:1] StatusLogger.java:47 log
> Pool NameActive   Pending  Completed   Blocked  All
> Time Blocked
> 2019-03-08 01:49:47,870 INFO  [ScheduledTasks:1] StatusLogger.java:51 log
> MutationStage 0 0   17641121 0
>0
> 2019-03-08 01:49:47,870 INFO  [ScheduledTasks:1] StatusLogger.java:51 log
> ViewMutationStage 0 0  0 0
>0
> 2019-03-08 01:49:47,870 INFO  [ScheduledTasks:1] StatusLogger.java:51 log
> ReadStage 0 06851090 0
>0
> 2019-03-08 01:49:47,870 INFO  [ScheduledTasks:1] StatusLogger.java:51 log
> RequestResponseStage  0 0   13646587 0
>0
> 2019-03-08 01:49:47,870 INFO  [ScheduledTasks:1] StatusLogger.java:51 log
> ReadRepairStage   0 0 352884 0
>0
> 2019-03-08 01:49:47,870 INFO  [ScheduledTasks:1] StatusLogger.java:51 log
> CounterMutationStage  0 0  0 0
>0
> 2019-03-08 01:49:47,870 INFO  [ScheduledTasks:1] StatusLogger.java:51 log
> MiscStage 0 0  0 0
>0
> 2019-03-08 01:49:47,870 INFO  [ScheduledTasks:1] StatusLogger.java:51 log
> CompactionExecutor0 0 882478 0
>0
> 2019-03-08 01:49:47,871 INFO  [ScheduledTasks:1] StatusLogger.java:51 log
> MemtableReclaimMemory 0 0   4101 0
>0
> 2019-03-08 01:49:47,871 INFO  [ScheduledTasks:1] StatusLogger.java:51 log
> PendingRangeCalculator0 0  7 0
>0
> 2019-03-08 01:49:47,871 INFO  [ScheduledTasks:1] StatusLogger.java:51 log
> GossipStage   0 04399705 0
>0
> 2019-03-08 01:49:47,871 INFO  [ScheduledTasks:1] StatusLogger.java:51 log
> SecondaryIndexManagement  0 0  0 0
>0
> 2019-03-08 01:49:47,871 INFO  [ScheduledTasks:1] StatusLogger.java:51 log
> HintsDispatcher   0 0   2165 0
>0
> 2019-03-08 01:49:47,871 INFO  [ScheduledTasks:1] StatusLogger.java:51 log
> MigrationStage0 0 50 0
>0
> 2019-03-08 01:49:47,871 INFO  [ScheduledTasks:1] StatusLogger.java:51 log
> MemtablePostFlush 0 0   4393 0
>0
> 2019-03-08 01:49:47,872 INFO  [ScheduledTasks:1] StatusLogger.java:51 log
> PerDiskMemtableFlushWriter_0 0 0   4097 0
>0
> 2019-03-08 01:49:47,872 INFO  [ScheduledTasks:1] StatusLogger.java:51 log
> ValidationExecutor0 0   1565 0
>0
> 2019-03-08 01:49:47,872 INFO  [ScheduledTasks:1] StatusLogger.java:51 log
> Sampler   0 0  0 0
>0
> 2019-03-08 01:49:47,872 INFO  [ScheduledTasks:1] StatusLogger.java:51 log
> MemtableFlushWriter   0 0   4101 0
>0
> 2019-03-08 01:49:47,872 INFO  [ScheduledTasks:1] StatusLogger.java:51 log
> InternalResponseStage 0 0 121813 0
>0
> 2019-03-08 01:49:47,872 INFO  [ScheduledTasks:1] StatusLogger.java:51 log
> AntiEntropyStage  0 0   6997 0
>0
> 2019-03-08 01:49:47,873 INFO  [ScheduledTasks:1] StatusLogger.java:51 log
> CacheCleanupExecutor  0 0 33 0
>0
> 2019-03-08 01:49:47,873 INFO  [ScheduledTasks:1] StatusLogger.java:51 log
> Native-Transport-Requests 4 0  364526951 0
>0
> 2019-03-08 01:49:47,873 INFO  [ScheduledTasks:1] StatusLogger.java:61 log
> CompactionManager 0 0
> 2019-03-08 01:49:47,873 INFO  [ScheduledTasks:1] StatusLogger.java:73 log
> MessagingServicen/a   2/0
> 2019-03-08 01:49:47,873 INFO  [ScheduledTasks:1] StatusLogger.java:83 log
> Cache Type Size