[jira] [Commented] (CASSANDRA-7467) flood of setting live ratio to maximum of 64 from repair
[
https://issues.apache.org/jira/browse/CASSANDRA-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088437#comment-14088437
]
graham sanderson commented on CASSANDRA-7467:
-
I can see where many of them are coming from (prior to this change but with
2.0.9 they started appearing)
{code}
2014-08-06 04:02:34,026 INFO [MemoryMeter:1] 2014-08-06 04:02:34,026
Memtable.java (line 481) CFS(Keyspace='system', ColumnFamily='schema_triggers')
liveRatio is 64.0 (just-counted was 64.0). calculation took 0ms for 0 cells
2014-08-06 04:02:34,069 INFO [MemoryMeter:1] 2014-08-06 04:02:34,069
Memtable.java (line 481) CFS(Keyspace='system', ColumnFamily='schema_triggers')
liveRatio is 64.0 (just-counted was 64.0). calculation took 0ms for 0 cells
2014-08-06 04:02:34,117 INFO [InternalResponseStage:87] 2014-08-06
04:02:34,117 ColumnFamilyStore.java (line 794) Enqueuing flush of
Memtable-schema_triggers@1406319171(0/0 serialized/live bytes, 3 ops)
2014-08-06 04:02:34,118 INFO [FlushWriter:41] 2014-08-06 04:02:34,118
Memtable.java (line 363) Writing Memtable-schema_triggers@1406319171(0/0
serialized/live bytes, 3 ops)
2014-08-06 04:02:34,124 INFO [FlushWriter:41] 2014-08-06 04:02:34,124
Memtable.java (line 403) Completed flushing
/data/6/cassandra/system/schema_triggers/system-schema_triggers-jb-60-Data.db
(129 bytes) for commitlog position ReplayPosition(segmentId=1407287842775,
position=7415363)
2014-08-06 04:02:34,126 INFO [CompactionExecutor:185] 2014-08-06 04:02:34,126
CompactionTask.java (line 115) Compacting
[SSTableReader(path='/data/1/cassandra/system/schema_triggers/system-schema_triggers-jb-59-Data.db'),
SSTableReader(path='/data/6/cassandra/system/schema_triggers/system-schema_triggers-jb-60-Data.db'),
SSTableReader(path='/data/6/cassandra/system/schema_triggers/system-schema_triggers-jb-58-Data.db'),
SSTableReader(path='/data/1/cassandra/system/schema_triggers/system-schema_triggers-jb-57-Data.db')]
2014-08-06 04:02:34,796 INFO [MemoryMeter:1] 2014-08-06 04:02:34,796
Memtable.java (line 481) CFS(Keyspace='system', ColumnFamily='schema_triggers')
liveRatio is 64.0 (just-counted was 64.0). calculation took 0ms for 0 cells
2014-08-06 04:02:34,844 INFO [MemoryMeter:1] 2014-08-06 04:02:34,844
Memtable.java (line 481) CFS(Keyspace='system', ColumnFamily='schema_triggers')
liveRatio is 64.0 (just-counted was 64.0). calculation took 0ms for 0 cells
2014-08-06 04:02:34,896 INFO [InternalResponseStage:88] 2014-08-06
04:02:34,896 ColumnFamilyStore.java (line 794) Enqueuing flush of
Memtable-schema_triggers@349252843(0/0 serialized/live bytes, 3 ops)
2014-08-06 04:02:34,897 INFO [FlushWriter:38] 2014-08-06 04:02:34,897
Memtable.java (line 363) Writing Memtable-schema_triggers@349252843(0/0
serialized/live bytes, 3 ops)
2014-08-06 04:02:34,903 INFO [FlushWriter:38] 2014-08-06 04:02:34,903
Memtable.java (line 403) Completed flushing
/data/1/cassandra/system/schema_triggers/system-schema_triggers-jb-61-Data.db
(129 bytes) for commitlog position ReplayPosition(segmentId=1407287842775,
position=7995460)
2014-08-06 04:12:47,723 INFO [MemoryMeter:1] 2014-08-06 04:12:47,723
Memtable.java (line 481) CFS(Keyspace='system', ColumnFamily='schema_triggers')
liveRatio is 64.0 (just-counted was 64.0). calculation took 0ms for 0 cells
2014-08-06 04:12:47,749 INFO [MemoryMeter:1] 2014-08-06 04:12:47,749
Memtable.java (line 481) CFS(Keyspace='system', ColumnFamily='schema_triggers')
liveRatio is 64.0 (just-counted was 64.0). calculation took 0ms for 0 cells
2014-08-06 04:12:47,815 INFO [InternalResponseStage:101] 2014-08-06
04:12:47,815 ColumnFamilyStore.java (line 794) Enqueuing flush of
Memtable-schema_triggers@1809531694(0/0 serialized/live bytes, 3 ops)
2014-08-06 04:12:47,816 INFO [FlushWriter:48] 2014-08-06 04:12:47,816
Memtable.java (line 363) Writing Memtable-schema_triggers@1809531694(0/0
serialized/live bytes, 3 ops)
2014-08-06 04:12:47,821 INFO [FlushWriter:48] 2014-08-06 04:12:47,821
Memtable.java (line 403) Completed flushing
/data/1/cassandra/system/schema_triggers/system-schema_triggers-jb-62-Data.db
(129 bytes) for commitlog position ReplayPosition(segmentId=1407287842777,
position=23729801)
2014-08-06 04:12:49,257 INFO [MemoryMeter:1] 2014-08-06 04:12:49,257
Memtable.java (line 481) CFS(Keyspace='system', ColumnFamily='schema_triggers')
liveRatio is 64.0 (just-counted was 64.0). calculation took 0ms for 0 cells
2014-08-06 04:12:49,301 INFO [MemoryMeter:1] 2014-08-06 04:12:49,301
Memtable.java (line 481) CFS(Keyspace='system', ColumnFamily='schema_triggers')
liveRatio is 64.0 (just-counted was 64.0). calculation took 0ms for 0 cells
2014-08-06 04:12:49,349 INFO [InternalResponseStage:103] 2014-08-06
04:12:49,349 ColumnFamilyStore.java (line 794) Enqueuing flush of
Memtable-schema_triggers@91416114(0/0 serialized/live bytes, 3 ops)
2014-08-06 04:12:49,350
[jira] [Commented] (CASSANDRA-7467) flood of setting live ratio to maximum of 64 from repair
[ https://issues.apache.org/jira/browse/CASSANDRA-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14077859#comment-14077859 ] Michael Shuler commented on CASSANDRA-7467: --- [~enigmacurry] could you see if this can be reproduced? Thanks! flood of setting live ratio to maximum of 64 from repair -- Key: CASSANDRA-7467 URL: https://issues.apache.org/jira/browse/CASSANDRA-7467 Project: Cassandra Issue Type: Bug Components: Core Reporter: Jackson Chung Fix For: 2.0.10 we are on 2.0.8 running with repair -pr -local KS, all nodes on i2.2x (60G ram);, with setting 8G of heap. Using java 8. (key cache size is 1G) On occasion, when repair is run, the C* that run the repair, or another node in the cluster, or both, run into a bad state with the system.log just printing setting live ratio to maximum of 64 forever every split seconds. It usually happens when repairing one of the larger/wider CF. WARN [MemoryMeter:1] 2014-06-28 09:13:24,540 Memtable.java (line 470) setting live ratio to maximum of 64.0 instead of Infinity INFO [MemoryMeter:1] 2014-06-28 09:13:24,540 Memtable.java (line 481) CFS(Keyspace='RIQ', ColumnFamily='MemberTimeline') liveRatio is 64.0 (just-counted was 64.0). calculation took 0ms for 0 cells Table: MemberTimeline SSTable count: 13 Space used (live), bytes: 17644018786 ... Compacted partition minimum bytes: 30 Compacted partition maximum bytes: 464228842 Compacted partition mean bytes: 54578 Just to give an idea of how bad this is, the log file is set to rotate 50 times with 21M each. In less than 15 minutes, all the logs are filled up with just that log. C* is not responding, and can't be killed normally. Only way is to kill -9 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7467) flood of setting live ratio to maximum of 64 from repair
[ https://issues.apache.org/jira/browse/CASSANDRA-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14047150#comment-14047150 ] Jackson Chung commented on CASSANDRA-7467: -- please ignore prev comment. 2 nodes misbehaved over night while repair was running (on another node), and crontab flush is already disabled. flood of setting live ratio to maximum of 64 from repair -- Key: CASSANDRA-7467 URL: https://issues.apache.org/jira/browse/CASSANDRA-7467 Project: Cassandra Issue Type: Bug Reporter: Jackson Chung we are on 2.0.8 running with repair -pr -local KS, all nodes on i2.2x (60G ram);, with setting 8G of heap. Using java 8. (key cache size is 1G) On occasion, when repair is run, the C* that run the repair, or another node in the cluster, or both, run into a bad state with the system.log just printing setting live ratio to maximum of 64 forever every split seconds. It usually happens when repairing one of the larger/wider CF. WARN [MemoryMeter:1] 2014-06-28 09:13:24,540 Memtable.java (line 470) setting live ratio to maximum of 64.0 instead of Infinity INFO [MemoryMeter:1] 2014-06-28 09:13:24,540 Memtable.java (line 481) CFS(Keyspace='RIQ', ColumnFamily='MemberTimeline') liveRatio is 64.0 (just-counted was 64.0). calculation took 0ms for 0 cells Table: MemberTimeline SSTable count: 13 Space used (live), bytes: 17644018786 ... Compacted partition minimum bytes: 30 Compacted partition maximum bytes: 464228842 Compacted partition mean bytes: 54578 Just to give an idea of how bad this is, the log file is set to rotate 50 times with 21M each. In less than 15 minutes, all the logs are filled up with just that log. C* is not responding, and can't be killed normally. Only way is to kill -9 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7467) flood of setting live ratio to maximum of 64 from repair
[ https://issues.apache.org/jira/browse/CASSANDRA-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14046958#comment-14046958 ] Aleksey Yeschenko commented on CASSANDRA-7467: -- Pretty sure this is a duplicate of CASSANDRA-7401. Now I'm really curious where those empty CFs are coming from, though. flood of setting live ratio to maximum of 64 from repair -- Key: CASSANDRA-7467 URL: https://issues.apache.org/jira/browse/CASSANDRA-7467 Project: Cassandra Issue Type: Bug Reporter: Jackson Chung we are on 2.0.8 running with repair -pr -local KS, all nodes on i2.2x (60G ram);, with setting 8G of heap. Using java 8. (key cache size is 1G) On occasion, when repair is run, the C* that run the repair, or another node in the cluster, or both, run into a bad state with the system.log just printing setting live ratio to maximum of 64 forever every split seconds. It usually happens when repairing one of the larger/wider CF. WARN [MemoryMeter:1] 2014-06-28 09:13:24,540 Memtable.java (line 470) setting live ratio to maximum of 64.0 instead of Infinity INFO [MemoryMeter:1] 2014-06-28 09:13:24,540 Memtable.java (line 481) CFS(Keyspace='RIQ', ColumnFamily='MemberTimeline') liveRatio is 64.0 (just-counted was 64.0). calculation took 0ms for 0 cells Table: MemberTimeline SSTable count: 13 Space used (live), bytes: 17644018786 ... Compacted partition minimum bytes: 30 Compacted partition maximum bytes: 464228842 Compacted partition mean bytes: 54578 Just to give an idea of how bad this is, the log file is set to rotate 50 times with 21M each. In less than 15 minutes, all the logs are filled up with just that log. C* is not responding, and can't be killed normally. Only way is to kill -9 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7467) flood of setting live ratio to maximum of 64 from repair
[ https://issues.apache.org/jira/browse/CASSANDRA-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14046975#comment-14046975 ] Jackson Chung commented on CASSANDRA-7467: -- We have some cron job that periodically run flush on the entire keyspace, so could be that flood of setting live ratio to maximum of 64 from repair -- Key: CASSANDRA-7467 URL: https://issues.apache.org/jira/browse/CASSANDRA-7467 Project: Cassandra Issue Type: Bug Reporter: Jackson Chung we are on 2.0.8 running with repair -pr -local KS, all nodes on i2.2x (60G ram);, with setting 8G of heap. Using java 8. (key cache size is 1G) On occasion, when repair is run, the C* that run the repair, or another node in the cluster, or both, run into a bad state with the system.log just printing setting live ratio to maximum of 64 forever every split seconds. It usually happens when repairing one of the larger/wider CF. WARN [MemoryMeter:1] 2014-06-28 09:13:24,540 Memtable.java (line 470) setting live ratio to maximum of 64.0 instead of Infinity INFO [MemoryMeter:1] 2014-06-28 09:13:24,540 Memtable.java (line 481) CFS(Keyspace='RIQ', ColumnFamily='MemberTimeline') liveRatio is 64.0 (just-counted was 64.0). calculation took 0ms for 0 cells Table: MemberTimeline SSTable count: 13 Space used (live), bytes: 17644018786 ... Compacted partition minimum bytes: 30 Compacted partition maximum bytes: 464228842 Compacted partition mean bytes: 54578 Just to give an idea of how bad this is, the log file is set to rotate 50 times with 21M each. In less than 15 minutes, all the logs are filled up with just that log. C* is not responding, and can't be killed normally. Only way is to kill -9 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7467) flood of setting live ratio to maximum of 64 from repair
[ https://issues.apache.org/jira/browse/CASSANDRA-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14047034#comment-14047034 ] Jackson Chung commented on CASSANDRA-7467: -- disabled the flush keyspace cron job, it does seem help. However, during the repair those lines would still be logged frequently (due to stream from the repair?). But at least it is not in a infinite-loop style. flood of setting live ratio to maximum of 64 from repair -- Key: CASSANDRA-7467 URL: https://issues.apache.org/jira/browse/CASSANDRA-7467 Project: Cassandra Issue Type: Bug Reporter: Jackson Chung we are on 2.0.8 running with repair -pr -local KS, all nodes on i2.2x (60G ram);, with setting 8G of heap. Using java 8. (key cache size is 1G) On occasion, when repair is run, the C* that run the repair, or another node in the cluster, or both, run into a bad state with the system.log just printing setting live ratio to maximum of 64 forever every split seconds. It usually happens when repairing one of the larger/wider CF. WARN [MemoryMeter:1] 2014-06-28 09:13:24,540 Memtable.java (line 470) setting live ratio to maximum of 64.0 instead of Infinity INFO [MemoryMeter:1] 2014-06-28 09:13:24,540 Memtable.java (line 481) CFS(Keyspace='RIQ', ColumnFamily='MemberTimeline') liveRatio is 64.0 (just-counted was 64.0). calculation took 0ms for 0 cells Table: MemberTimeline SSTable count: 13 Space used (live), bytes: 17644018786 ... Compacted partition minimum bytes: 30 Compacted partition maximum bytes: 464228842 Compacted partition mean bytes: 54578 Just to give an idea of how bad this is, the log file is set to rotate 50 times with 21M each. In less than 15 minutes, all the logs are filled up with just that log. C* is not responding, and can't be killed normally. Only way is to kill -9 -- This message was sent by Atlassian JIRA (v6.2#6252)
