[jira] [Commented] (CASSANDRA-8641) Repair causes a large number of tiny SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-8641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14627072#comment-14627072 ] kris commented on CASSANDRA-8641: - The below result is related to the comment posted by Kenneth Falibus. The total count of the SStables on this node for x columnfamily is "40940". SSTables of size <1kb for x CF= 11425 SSTables of size >1kb and <100kbfor x CF= 22872 This happens on just a couple nodes of the 24 nodes. Read Count: 0 Read Latency: NaN ms. Write Count: 1996761 Write Latency: 0.08124631640942506 ms. Pending Tasks: 51 SSTable count: 40940 Space used (live), bytes: 258350112587 Space used (total), bytes: 258350450394 Off heap memory used (total), bytes: 3424782751 SSTable Compression Ratio: 0.6112709276839385 Number of keys (estimate): 1567894400 Memtable cell count: 2918 Memtable data size, bytes: 2001141 Memtable switch count: 176 Local read count: 0 Local read latency: 0.000 ms Local write count: 1996761 Local write latency: 0.000 ms Pending tasks: 51 Bloom filter false positives: 0 Bloom filter false ratio: 0.0 Bloom filter space used, bytes: 2779811296 Bloom filter off heap memory used, bytes: 2779483776 Index summary off heap memory used, bytes: 603271015 Compression metadata off heap memory used, bytes: 42027960 Compacted partition minimum bytes: 43 Compacted partition maximum bytes: 785939 Compacted partition mean bytes: 240 Average live cells per slice (last five minutes): 0.0 Average tombstones per slice (last five minutes): 0.0 > Repair causes a large number of tiny SSTables > - > > Key: CASSANDRA-8641 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8641 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 14.04 >Reporter: Flavien Charlon > Fix For: 2.1.3 > > > I have a 3 nodes cluster with RF = 3, quad core and 32 GB or RAM. I am > running 2.1.2 with all the default settings. I'm seeing some strange > behaviors during incremental repair (under write load). > Taking the example of one particular column family, before running an > incremental repair, I have about 13 SSTables. After finishing the incremental > repair, I have over 114000 SSTables. > {noformat} > Table: customers > SSTable count: 114688 > Space used (live): 97203707290 > Space used (total): 99175455072 > Space used by snapshots (total): 0 > SSTable Compression Ratio: 0.28281112416526505 > Memtable cell count: 0 > Memtable data size: 0 > Memtable switch count: 1069 > Local read count: 0 > Local read latency: NaN ms > Local write count: 11548705 > Local write latency: 0.030 ms > Pending flushes: 0 > Bloom filter false positives: 0 > Bloom filter false ratio: 0.0 > Bloom filter space used: 144145152 > Compacted partition minimum bytes: 311 > Compacted partition maximum bytes: 1996099046 > Compacted partition mean bytes: 3419 > Average live cells per slice (last five minutes): 0.0 > Maximum live cells per slice (last five minutes): 0.0 > Average tombstones per slice (last five minutes): 0.0 > Maximum tombstones per slice (last five minutes): 0.0 > {noformat} > Looking at the logs during the repair, it seems Cassandra is struggling to > compact minuscule memtables (often just a few kilobytes): > {noformat} > INFO [CompactionExecutor:337] 2015-01-17 01:44:27,011 > CompactionTask.java:251 - Compacted 32 sstables to > [/mnt/data/cassandra/data/business/customers-d9d42d209ccc11e48ca54553c90a9d45/business-customers-ka-228341,]. > 8,332 bytes to 6,547 (~78% of original) in 80,476ms = 0.78MB/s. 32 > total partitions merged to 32. Partition merge counts were {1:32, } > INFO [CompactionExecutor:337] 2015-01-17 01:45:35,519 > CompactionTask.java:251 - Compacted 32 sstables to > [/mnt/data/cassandra/data/business/customers-d9d42d209ccc11e48ca54553c90a9d45/business-customers-ka-229348,]. > 8,384 bytes to 6,563 (~78% of original) in 6,880ms = 0.000910MB/s. 32 > total partitions merged to 32. Partition merge counts were {1:32, } > INFO [CompactionExecutor:339] 2015-01-17 01:47:46,475 > CompactionTask.java:251 - Compacted 32 sstables to > [/mnt/data/cassandra/data/business/customers-d9d42d209ccc11e48ca54553c90a9d45/business-customers-ka-229351,]. > 8,423 bytes to 6,401 (~75% of original) in 10,416ms = 0.000586MB/s. 32 > total partitions merged to 32. Partition merge counts were {1:32, } > {noform
[jira] [Commented] (CASSANDRA-8422) cassandra won't start up due to "Unable to gossip with any seeds" on the decommissioned node
[ https://issues.apache.org/jira/browse/CASSANDRA-8422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14293192#comment-14293192 ] kris commented on CASSANDRA-8422: - we are using dse 4.6 and facing the same issue. > cassandra won't start up due to "Unable to gossip with any seeds" on the > decommissioned node > > > Key: CASSANDRA-8422 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8422 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Masashi Ozawa > > - 2-node > * nodeA - seed > * nodeB > 1. decommission nodeB from the cluster with nodetool >when it's finished, kill cassandra process on nodeB > 2. delete data from commit/cache/data directories on nodeB > 3. try to start cassandra on nodeB (first time) >=> FAILED with "Unable to gossip with any seeds" > 4. try to start cassandra on nodeB (second time) > => OK > It was not a one-time shot. I tried it a several times and encountered the > same issue for some reason. > {code} > ERROR [main] 2014-11-27 18:44:55,017 CassandraDaemon.java (line 513) > Exception encountered during startup > java.lang.RuntimeException: Unable to gossip with any seeds > at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1211) > at > org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:445) > at > org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:659) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:611) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:503) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585) > INFO [StorageServiceShutdownHook] 2014-11-27 18:44:55,076 Gossiper.java > (line 1307) Announcing shutdown > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7350) Decommissioning nodes borks the seed node - can't add additional nodes
[ https://issues.apache.org/jira/browse/CASSANDRA-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14293197#comment-14293197 ] kris commented on CASSANDRA-7350: - we are facing the same issue too. > Decommissioning nodes borks the seed node - can't add additional nodes > -- > > Key: CASSANDRA-7350 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7350 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu using the auto-clustering AMI >Reporter: Steven Lowenthal >Assignee: Shawn Kumar >Priority: Minor > Labels: qa-resolved > Fix For: 2.0.9 > > > 1) Launch a 4 node cluster - I used the auto-clustering AMI (you get nodes > 0-3) > 2) decommission that last 2 nodes (nodes , leaving a 2 node cluster) > 3) wipe the data directories from node 2 > 4) bootstrap node2 - it won't join "unable to gossip with any seeds". > If you bootstrap the node a second time, it will join. However if you try to > bootstrap node 3, it will also fail. > I discovered that bouncing the seed node fixes the problem. I think it > cropped up in 2.0.7. > Error: > ERROR [main] 2014-06-03 21:52:46,649 CassandraDaemon.java (line 497) > Exception encountered during startup > java.lang.RuntimeException: Unable to gossip with any seeds > at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1193) > at > org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:447) > at > org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:656) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:612) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:505) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:362) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:480) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:569) > ERROR [StorageServiceShutdownHook] 2014-06-03 21:52:46,741 > CassandraDaemon.java (line 198) Exception in thread Thread[StorageServi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8422) cassandra won't start up due to "Unable to gossip with any seeds" on the decommissioned node
[ https://issues.apache.org/jira/browse/CASSANDRA-8422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14293210#comment-14293210 ] kris commented on CASSANDRA-8422: - the cassandra version is 2.0.11.83 > cassandra won't start up due to "Unable to gossip with any seeds" on the > decommissioned node > > > Key: CASSANDRA-8422 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8422 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Masashi Ozawa > > - 2-node > * nodeA - seed > * nodeB > 1. decommission nodeB from the cluster with nodetool >when it's finished, kill cassandra process on nodeB > 2. delete data from commit/cache/data directories on nodeB > 3. try to start cassandra on nodeB (first time) >=> FAILED with "Unable to gossip with any seeds" > 4. try to start cassandra on nodeB (second time) > => OK > It was not a one-time shot. I tried it a several times and encountered the > same issue for some reason. > {code} > ERROR [main] 2014-11-27 18:44:55,017 CassandraDaemon.java (line 513) > Exception encountered during startup > java.lang.RuntimeException: Unable to gossip with any seeds > at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1211) > at > org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:445) > at > org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:659) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:611) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:503) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585) > INFO [StorageServiceShutdownHook] 2014-11-27 18:44:55,076 Gossiper.java > (line 1307) Announcing shutdown > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[ https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14484169#comment-14484169 ] kris commented on CASSANDRA-8696: - faced the same issue with dse 4.6 and cassandra version 2.0.11.83 > nodetool repair on cassandra 2.1.2 keyspaces return > java.lang.RuntimeException: Could not create snapshot > - > > Key: CASSANDRA-8696 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8696 > Project: Cassandra > Issue Type: Bug >Reporter: Jeff Liu >Assignee: Yuki Morishita > Fix For: 2.1.5 > > > When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra > throw java exceptions: cannot create snapshot. > the error log from system.log: > {noformat} > INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815 > StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227 > ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5 > files(632105 bytes) > INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046 > StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227] > Session with /10.97.9.110 is complete > INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046 > StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227] > All sessions completed > INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047 > StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227] > streaming task succeed, returning response to /10.98.194.68 > INFO [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 - > [Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for > Repair > INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065 > StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227] > Starting streaming to /10.66.187.201 > INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070 > StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227, > ID#0] Beginning stream session with /10.66.187.201 > INFO [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465 > StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227 > ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5 > files(632105 bytes) > INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,971 > StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227] > Session with /10.66.187.201 is complete > INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972 > StreamResultFuture.java:212 - [Stream #692c6270-a692-11e4-9973-070e938df227] > All sessions completed > INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972 > StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227] > streaming task succeed, returning response to /10.98.194.68 > ERROR [RepairJobTask:1] 2015-01-28 02:07:39,444 RepairJob.java:127 - Error > occurred during snapshot phase > java.lang.RuntimeException: Could not create snapshot at /10.97.9.110 > at > org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > ~[na:1.7.0_45] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ~[na:1.7.0_45] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_45] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_45] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] > INFO [AntiEntropySessions:6] 2015-01-28 02:07:39,445 RepairSession.java:260 > - [repair #6f85e740-a692-11e4-9973-070e938df227] new session: will sync > /10.98.194.68, /10.66.187.201, /10.226.218.135 on range > (12817179804668051873746972069086 > 2638799,12863540308359254031520865977436165] for events.[bigint0text, > bigint0boolean, bigint0int, dataset_catalog, column_categories, > bigint0double, bigint0bigint] > ERROR [AntiEntropySessions:5] 2015-01-28 02:07:39,445 RepairSession.java:303 > - [repair #685e3d00-a692-11e4-9973-070e938df227] session completed with the > following error > java.io.IOException: Failed during snapshot creation. > at > org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:128) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at com.google.common.util.concurrent.Futures$4.run(
[jira] [Comment Edited] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[ https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14484169#comment-14484169 ] kris edited comment on CASSANDRA-8696 at 4/7/15 10:00 PM: -- faced the same issue with dse 4.6 and cassandra version 2.0.11.83. Have been running "repair" with "opscenter" was (Author: kris88): faced the same issue with dse 4.6 and cassandra version 2.0.11.83 > nodetool repair on cassandra 2.1.2 keyspaces return > java.lang.RuntimeException: Could not create snapshot > - > > Key: CASSANDRA-8696 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8696 > Project: Cassandra > Issue Type: Bug >Reporter: Jeff Liu >Assignee: Yuki Morishita > Fix For: 2.1.5 > > > When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra > throw java exceptions: cannot create snapshot. > the error log from system.log: > {noformat} > INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815 > StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227 > ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5 > files(632105 bytes) > INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046 > StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227] > Session with /10.97.9.110 is complete > INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046 > StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227] > All sessions completed > INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047 > StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227] > streaming task succeed, returning response to /10.98.194.68 > INFO [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 - > [Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for > Repair > INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065 > StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227] > Starting streaming to /10.66.187.201 > INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070 > StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227, > ID#0] Beginning stream session with /10.66.187.201 > INFO [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465 > StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227 > ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5 > files(632105 bytes) > INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,971 > StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227] > Session with /10.66.187.201 is complete > INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972 > StreamResultFuture.java:212 - [Stream #692c6270-a692-11e4-9973-070e938df227] > All sessions completed > INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972 > StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227] > streaming task succeed, returning response to /10.98.194.68 > ERROR [RepairJobTask:1] 2015-01-28 02:07:39,444 RepairJob.java:127 - Error > occurred during snapshot phase > java.lang.RuntimeException: Could not create snapshot at /10.97.9.110 > at > org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > ~[na:1.7.0_45] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ~[na:1.7.0_45] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_45] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_45] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] > INFO [AntiEntropySessions:6] 2015-01-28 02:07:39,445 RepairSession.java:260 > - [repair #6f85e740-a692-11e4-9973-070e938df227] new session: will sync > /10.98.194.68, /10.66.187.201, /10.226.218.135 on range > (12817179804668051873746972069086 > 2638799,12863540308359254031520865977436165] for events.[bigint0text, > bigint0boolean, bigint0int, dataset_catalog, column_categories, > bigint0double, bigint0bigint] > ERROR [AntiEntropySessions:5] 2015-01-28 02:07:39,445 RepairSession.java:303 > - [repair #685e3d00-a692-11e4-9973-070e938df227] session completed with the > following error > java.io.IOException: Failed during snapshot creation. > at > org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344) > ~[apache-cassandra-2.1.2.jar:2.1.2] >