[jira] [Commented] (CASSANDRA-8641) Repair causes a large number of tiny SSTables
[
https://issues.apache.org/jira/browse/CASSANDRA-8641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14627072#comment-14627072
]
kris commented on CASSANDRA-8641:
-
The below result is related to the comment posted by Kenneth Falibus. The total
count of the SStables on this node for x columnfamily is "40940".
SSTables of size <1kb for x CF= 11425
SSTables of size >1kb and <100kbfor x CF= 22872
This happens on just a couple nodes of the 24 nodes.
Read Count: 0
Read Latency: NaN ms.
Write Count: 1996761
Write Latency: 0.08124631640942506 ms.
Pending Tasks: 51
SSTable count: 40940
Space used (live), bytes: 258350112587
Space used (total), bytes: 258350450394
Off heap memory used (total), bytes: 3424782751
SSTable Compression Ratio: 0.6112709276839385
Number of keys (estimate): 1567894400
Memtable cell count: 2918
Memtable data size, bytes: 2001141
Memtable switch count: 176
Local read count: 0
Local read latency: 0.000 ms
Local write count: 1996761
Local write latency: 0.000 ms
Pending tasks: 51
Bloom filter false positives: 0
Bloom filter false ratio: 0.0
Bloom filter space used, bytes: 2779811296
Bloom filter off heap memory used, bytes: 2779483776
Index summary off heap memory used, bytes: 603271015
Compression metadata off heap memory used, bytes: 42027960
Compacted partition minimum bytes: 43
Compacted partition maximum bytes: 785939
Compacted partition mean bytes: 240
Average live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
> Repair causes a large number of tiny SSTables
> -
>
> Key: CASSANDRA-8641
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8641
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Ubuntu 14.04
>Reporter: Flavien Charlon
> Fix For: 2.1.3
>
>
> I have a 3 nodes cluster with RF = 3, quad core and 32 GB or RAM. I am
> running 2.1.2 with all the default settings. I'm seeing some strange
> behaviors during incremental repair (under write load).
> Taking the example of one particular column family, before running an
> incremental repair, I have about 13 SSTables. After finishing the incremental
> repair, I have over 114000 SSTables.
> {noformat}
> Table: customers
> SSTable count: 114688
> Space used (live): 97203707290
> Space used (total): 99175455072
> Space used by snapshots (total): 0
> SSTable Compression Ratio: 0.28281112416526505
> Memtable cell count: 0
> Memtable data size: 0
> Memtable switch count: 1069
> Local read count: 0
> Local read latency: NaN ms
> Local write count: 11548705
> Local write latency: 0.030 ms
> Pending flushes: 0
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 144145152
> Compacted partition minimum bytes: 311
> Compacted partition maximum bytes: 1996099046
> Compacted partition mean bytes: 3419
> Average live cells per slice (last five minutes): 0.0
> Maximum live cells per slice (last five minutes): 0.0
> Average tombstones per slice (last five minutes): 0.0
> Maximum tombstones per slice (last five minutes): 0.0
> {noformat}
> Looking at the logs during the repair, it seems Cassandra is struggling to
> compact minuscule memtables (often just a few kilobytes):
> {noformat}
> INFO [CompactionExecutor:337] 2015-01-17 01:44:27,011
> CompactionTask.java:251 - Compacted 32 sstables to
> [/mnt/data/cassandra/data/business/customers-d9d42d209ccc11e48ca54553c90a9d45/business-customers-ka-228341,].
> 8,332 bytes to 6,547 (~78% of original) in 80,476ms = 0.78MB/s. 32
> total partitions merged to 32. Partition merge counts were {1:32, }
> INFO [CompactionExecutor:337] 2015-01-17 01:45:35,519
> CompactionTask.java:251 - Compacted 32 sstables to
> [/mnt/data/cassandra/data/business/customers-d9d42d209ccc11e48ca54553c90a9d45/business-customers-ka-229348,].
> 8,384 bytes to 6,563 (~78% of original) in 6,880ms = 0.000910MB/s. 32
> total partitions merged to 32. Partition merge counts were {1:32, }
> INFO [CompactionExecutor:339] 2015-01-17 01:47:46,475
> CompactionTask.java:251 - Compacted 32 sstables to
> [/mnt/data/cassandra/data/business/customers-d9d42d209ccc11e48ca54553c90a9d45/business-customers-ka-229351,].
> 8,423 bytes to 6,401 (~75% of original) in 10,416ms = 0.000586MB/s. 32
> total partitions merged to 32. Partition merge counts were {1:32, }
> {noform
[jira] [Commented] (CASSANDRA-8422) cassandra won't start up due to "Unable to gossip with any seeds" on the decommissioned node
[
https://issues.apache.org/jira/browse/CASSANDRA-8422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14293192#comment-14293192
]
kris commented on CASSANDRA-8422:
-
we are using dse 4.6 and facing the same issue.
> cassandra won't start up due to "Unable to gossip with any seeds" on the
> decommissioned node
>
>
> Key: CASSANDRA-8422
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8422
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Masashi Ozawa
>
> - 2-node
> * nodeA - seed
> * nodeB
> 1. decommission nodeB from the cluster with nodetool
>when it's finished, kill cassandra process on nodeB
> 2. delete data from commit/cache/data directories on nodeB
> 3. try to start cassandra on nodeB (first time)
>=> FAILED with "Unable to gossip with any seeds"
> 4. try to start cassandra on nodeB (second time)
> => OK
> It was not a one-time shot. I tried it a several times and encountered the
> same issue for some reason.
> {code}
> ERROR [main] 2014-11-27 18:44:55,017 CassandraDaemon.java (line 513)
> Exception encountered during startup
> java.lang.RuntimeException: Unable to gossip with any seeds
> at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1211)
> at
> org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:445)
> at
> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:659)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:611)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:503)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
> INFO [StorageServiceShutdownHook] 2014-11-27 18:44:55,076 Gossiper.java
> (line 1307) Announcing shutdown
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7350) Decommissioning nodes borks the seed node - can't add additional nodes
[ https://issues.apache.org/jira/browse/CASSANDRA-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14293197#comment-14293197 ] kris commented on CASSANDRA-7350: - we are facing the same issue too. > Decommissioning nodes borks the seed node - can't add additional nodes > -- > > Key: CASSANDRA-7350 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7350 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu using the auto-clustering AMI >Reporter: Steven Lowenthal >Assignee: Shawn Kumar >Priority: Minor > Labels: qa-resolved > Fix For: 2.0.9 > > > 1) Launch a 4 node cluster - I used the auto-clustering AMI (you get nodes > 0-3) > 2) decommission that last 2 nodes (nodes , leaving a 2 node cluster) > 3) wipe the data directories from node 2 > 4) bootstrap node2 - it won't join "unable to gossip with any seeds". > If you bootstrap the node a second time, it will join. However if you try to > bootstrap node 3, it will also fail. > I discovered that bouncing the seed node fixes the problem. I think it > cropped up in 2.0.7. > Error: > ERROR [main] 2014-06-03 21:52:46,649 CassandraDaemon.java (line 497) > Exception encountered during startup > java.lang.RuntimeException: Unable to gossip with any seeds > at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1193) > at > org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:447) > at > org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:656) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:612) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:505) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:362) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:480) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:569) > ERROR [StorageServiceShutdownHook] 2014-06-03 21:52:46,741 > CassandraDaemon.java (line 198) Exception in thread Thread[StorageServi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8422) cassandra won't start up due to "Unable to gossip with any seeds" on the decommissioned node
[
https://issues.apache.org/jira/browse/CASSANDRA-8422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14293210#comment-14293210
]
kris commented on CASSANDRA-8422:
-
the cassandra version is 2.0.11.83
> cassandra won't start up due to "Unable to gossip with any seeds" on the
> decommissioned node
>
>
> Key: CASSANDRA-8422
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8422
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Masashi Ozawa
>
> - 2-node
> * nodeA - seed
> * nodeB
> 1. decommission nodeB from the cluster with nodetool
>when it's finished, kill cassandra process on nodeB
> 2. delete data from commit/cache/data directories on nodeB
> 3. try to start cassandra on nodeB (first time)
>=> FAILED with "Unable to gossip with any seeds"
> 4. try to start cassandra on nodeB (second time)
> => OK
> It was not a one-time shot. I tried it a several times and encountered the
> same issue for some reason.
> {code}
> ERROR [main] 2014-11-27 18:44:55,017 CassandraDaemon.java (line 513)
> Exception encountered during startup
> java.lang.RuntimeException: Unable to gossip with any seeds
> at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1211)
> at
> org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:445)
> at
> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:659)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:611)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:503)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
> INFO [StorageServiceShutdownHook] 2014-11-27 18:44:55,076 Gossiper.java
> (line 1307) Announcing shutdown
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14484169#comment-14484169
]
kris commented on CASSANDRA-8696:
-
faced the same issue with dse 4.6 and cassandra version 2.0.11.83
> nodetool repair on cassandra 2.1.2 keyspaces return
> java.lang.RuntimeException: Could not create snapshot
> -
>
> Key: CASSANDRA-8696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8696
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>Assignee: Yuki Morishita
> Fix For: 2.1.5
>
>
> When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra
> throw java exceptions: cannot create snapshot.
> the error log from system.log:
> {noformat}
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815
> StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5
> files(632105 bytes)
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> Session with /10.97.9.110 is complete
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> INFO [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 -
> [Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for
> Repair
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065
> StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Starting streaming to /10.66.187.201
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070
> StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227,
> ID#0] Beginning stream session with /10.66.187.201
> INFO [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465
> StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5
> files(632105 bytes)
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,971
> StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Session with /10.66.187.201 is complete
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamResultFuture.java:212 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> ERROR [RepairJobTask:1] 2015-01-28 02:07:39,444 RepairJob.java:127 - Error
> occurred during snapshot phase
> java.lang.RuntimeException: Could not create snapshot at /10.97.9.110
> at
> org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[na:1.7.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> INFO [AntiEntropySessions:6] 2015-01-28 02:07:39,445 RepairSession.java:260
> - [repair #6f85e740-a692-11e4-9973-070e938df227] new session: will sync
> /10.98.194.68, /10.66.187.201, /10.226.218.135 on range
> (12817179804668051873746972069086
> 2638799,12863540308359254031520865977436165] for events.[bigint0text,
> bigint0boolean, bigint0int, dataset_catalog, column_categories,
> bigint0double, bigint0bigint]
> ERROR [AntiEntropySessions:5] 2015-01-28 02:07:39,445 RepairSession.java:303
> - [repair #685e3d00-a692-11e4-9973-070e938df227] session completed with the
> following error
> java.io.IOException: Failed during snapshot creation.
> at
> org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:128)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at com.google.common.util.concurrent.Futures$4.run(
[jira] [Comment Edited] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14484169#comment-14484169
]
kris edited comment on CASSANDRA-8696 at 4/7/15 10:00 PM:
--
faced the same issue with dse 4.6 and cassandra version 2.0.11.83. Have been
running "repair" with "opscenter"
was (Author: kris88):
faced the same issue with dse 4.6 and cassandra version 2.0.11.83
> nodetool repair on cassandra 2.1.2 keyspaces return
> java.lang.RuntimeException: Could not create snapshot
> -
>
> Key: CASSANDRA-8696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8696
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>Assignee: Yuki Morishita
> Fix For: 2.1.5
>
>
> When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra
> throw java exceptions: cannot create snapshot.
> the error log from system.log:
> {noformat}
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815
> StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5
> files(632105 bytes)
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> Session with /10.97.9.110 is complete
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> INFO [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 -
> [Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for
> Repair
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065
> StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Starting streaming to /10.66.187.201
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070
> StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227,
> ID#0] Beginning stream session with /10.66.187.201
> INFO [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465
> StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5
> files(632105 bytes)
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,971
> StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Session with /10.66.187.201 is complete
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamResultFuture.java:212 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> ERROR [RepairJobTask:1] 2015-01-28 02:07:39,444 RepairJob.java:127 - Error
> occurred during snapshot phase
> java.lang.RuntimeException: Could not create snapshot at /10.97.9.110
> at
> org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[na:1.7.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> INFO [AntiEntropySessions:6] 2015-01-28 02:07:39,445 RepairSession.java:260
> - [repair #6f85e740-a692-11e4-9973-070e938df227] new session: will sync
> /10.98.194.68, /10.66.187.201, /10.226.218.135 on range
> (12817179804668051873746972069086
> 2638799,12863540308359254031520865977436165] for events.[bigint0text,
> bigint0boolean, bigint0int, dataset_catalog, column_categories,
> bigint0double, bigint0bigint]
> ERROR [AntiEntropySessions:5] 2015-01-28 02:07:39,445 RepairSession.java:303
> - [repair #685e3d00-a692-11e4-9973-070e938df227] session completed with the
> following error
> java.io.IOException: Failed during snapshot creation.
> at
> org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
>
