[jira] [Commented] (CASSANDRA-14495) Memory Leak /High Memory usage post 3.11.2 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661010#comment-16661010 ] Abdul Patel commented on CASSANDRA-14495: - i only see high heap memory usage nothing else, and when i searched for GCInspector i found the the listed message > Memory Leak /High Memory usage post 3.11.2 upgrade > -- > > Key: CASSANDRA-14495 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14495 > Project: Cassandra > Issue Type: Bug > Components: Metrics >Reporter: Abdul Patel >Priority: Major > Attachments: cas_heap.txt > > > Hi All, > > I recently upgraded my non prod cassandra cluster( 4 nodes single DC) from > 3.10 to 3.11.2 version. > No issues reported apart from only nodetool info reporting 80% usage . > I intially had 16GB memory on each node, later i bumped up to 20GB, and > rebooted all nodes. > Waited for an week and now again i have seen memory usage more than 80% , > 16GB + . > this means some memory leaks are happening over the time. > Any one has faced such issue or do we have any workaround ? my 3.11.2 version > upgrade rollout has been halted because of this bug. > === > ID : 65b64f5a-7fe6-4036-94c8-8da9c57718cc > Gossip active : true > Thrift active : true > Native Transport active: true > Load : 985.24 MiB > Generation No : 1526923117 > Uptime (seconds) : 1097684 > Heap Memory (MB) : 16875.64 / 20480.00 > Off Heap Memory (MB) : 20.42 > Data Center : DC7 > Rack : rac1 > Exceptions : 0 > Key Cache : entries 3569, size 421.44 KiB, capacity 100 MiB, > 7931933 hits, 8098632 requests, 0.979 recent hit rate, 14400 save period in > seconds > Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 > requests, NaN recent hit rate, 0 save period in seconds > Counter Cache : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 > requests, NaN recent hit rate, 7200 save period in seconds > Chunk Cache : entries 2361, size 147.56 MiB, capacity 3.97 GiB, > 2412803 misses, 72594047 requests, 0.967 recent hit rate, NaN microseconds > miss latency > Percent Repaired : 99.88086234106282% > Token : (invoke with -T/--tokens to see all 256 tokens) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14495) Memory Leak /High Memory usage post 3.11.2 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660666#comment-16660666 ] Abdul Patel commented on CASSANDRA-14495: - I see below GC inspector messages, can some help me undertsnad these ? INFO [Service Thread] 2018-10-22 11:10:22,178 GCInspector.java:284 - G1 Young Generation GC in 251ms. G1 Eden Space: 729808896 -> 0; G1 Old Gen: 7029653504 -> 7138390512; G1 Survivor Space: 109051904 -> 75497472; INFO [Service Thread] 2018-10-22 11:16:05,708 GCInspector.java:284 - G1 Young Generation GC in 209ms. G1 Eden Space: 729808896 -> 0; G1 Old Gen: 4353687552 -> 4483710984; INFO [Service Thread] 2018-10-22 11:16:08,759 GCInspector.java:284 - G1 Young Generation GC in 253ms. G1 Eden Space: 729808896 -> 0; G1 Old Gen: 4483710984 -> 4601151480; INFO [Service Thread] 2018-10-22 11:20:26,201 GCInspector.java:284 - G1 Young Generation GC in 217ms. G1 Eden Space: 729808896 -> 0; G1 Old Gen: 3602907144 -> 2015954944; INFO [Service Thread] 2018-10-22 11:20:34,467 GCInspector.java:284 - G1 Young Generation GC in 206ms. G1 Eden Space: 729808896 -> 0; G1 Old Gen: 1874853896 -> 2052537840; INFO [Service Thread] 2018-10-22 11:48:32,697 GCInspector.java:284 - G1 Young Generation GC in 573ms. G1 Eden Space: 10200547328 -> 0; G1 Old Gen: 35824 -> 0; G1 Survivor Space: 100663296 -> 939524096; INFO [Service Thread] 2018-10-22 11:48:38,038 GCInspector.java:284 - G1 Young Generation GC in 793ms. G1 Eden Space: 3078619136 -> 0; G1 Old Gen: 0 -> 907214328; G1 Survivor Space: 939524096 -> 394264576; INFO [Service Thread] 2018-10-22 11:48:39,133 GCInspector.java:284 - G1 Young Generation GC in 294ms. G1 Eden Space: 461373440 -> 0; G1 Old Gen: 907214328 -> 1291845632; G1 Survivor Space: 394264576 -> 75497472; INFO [Service Thread] 2018-10-22 11:49:12,405 GCInspector.java:284 - G1 Young Generation GC in 222ms. G1 Eden Space: 2181038080 -> 0; G1 Old Gen: 3481272304 -> 3677068784; G1 Survivor Space: 251658240 -> 243269632; INFO [Service Thread] 2018-10-22 11:49:34,485 GCInspector.java:284 - G1 Young Generation GC in 210ms. G1 Eden Space: 4085252096 -> 0; G1 Survivor Space: 67108864 -> 234881024; INFO [Service Thread] 2018-10-22 11:49:41,027 GCInspector.java:284 - G1 Young Generation GC in 208ms. G1 Eden Space: 2290089984 -> 0; G1 Old Gen: 4903141368 -> 5096079352; G1 Survivor Space: 192937984 -> 100663296; INFO [Service Thread] 2018-10-22 11:49:47,059 GCInspector.java:284 - G1 Young Generation GC in 229ms. G1 Eden Space: 2113929216 -> 0; G1 Old Gen: 5096079352 -> 5179965448; G1 Survivor Space: 100663296 -> 260046848; INFO [Service Thread] 2018-10-22 11:49:47,864 GCInspector.java:284 - G1 Young Generation GC in 240ms. G1 Eden Space: 595591168 -> 0; G1 Old Gen: 5179965448 -> 5456591864; G1 Survivor Space: 260046848 -> 41943040; INFO [Service Thread] 2018-10-22 11:51:55,126 GCInspector.java:284 - G1 Young Generation GC in 682ms. G1 Eden Space: 10208935936 -> 0; G1 Old Gen: 2657805472 -> 2663677936; G1 Survivor Space: 92274688 -> 830472192; INFO [Service Thread] 2018-10-22 11:52:02,632 GCInspector.java:284 - G1 Young Generation GC in 614ms. G1 Eden Space: 2558525440 -> 0; G1 Old Gen: 2663677936 -> 3467692024; G1 Survivor Space: 830472192 -> 318767104; INFO [Service Thread] 2018-10-22 11:52:04,595 GCInspector.java:284 - G1 Young Generation GC in 213ms. G1 Eden Space: 536870912 -> 0; G1 Old Gen: 3467692024 -> 3783262192; G1 Survivor Space: 318767104 -> 83886080; INFO [Service Thread] 2018-10-22 11:53:41,556 GCInspector.java:284 - G1 Young Generation GC in 279ms. G1 Eden Space: 10150215680 -> 0; G1 Old Gen: 3793081848 -> 3797276144; G1 Survivor Space: 150994944 -> 662700032; INFO [Service Thread] 2018-10-22 11:53:51,744 GCInspector.java:284 - G1 Young Generation GC in 521ms. G1 Eden Space: 7440695296 -> 0; G1 Old Gen: 3797276144 -> 3918572016; G1 Survivor Space: 662700032 -> 998244352; INFO [Service Thread] 2018-10-22 11:53:52,370 GCInspector.java:284 - G1 Young Generation GC in 589ms. G1 Eden Space: 8388608 -> 0; G1 Old Gen: 3918572016 -> 4907335664; G1 Survivor Space: 998244352 -> 33554432; INFO [Service Thread] 2018-10-22 11:54:31,296 GCInspector.java:284 - G1 Young Generation GC in 253ms. G1 Eden Space: 1937768448 -> 0; G1 Old Gen: 3246391280 -> 3661627384; G1 Survivor Space: 394264576 -> 100663296; INFO [Service Thread] 2018-10-22 11:54:42,711 GCInspector.java:284 - G1 Young Generation GC in 210ms. G1 Eden Space: 872415232 -> 0; G1 Old Gen: 3955228664 -> 4215275512; G1 Survivor Space: 260046848 -> 67108864; INFO [Service Thread] 2018-10-22 11:54:47,487 GCInspector.java:284 - G1 Young Generation GC in 207ms. G1 Eden Space: 3816816640 -> 0; G1 Survivor Space: 67108864 -> 260046848; INFO [Service Thread] 2018-10-22 11:54:48,615 GCInspector.java:284 - G1 Young Generation GC in 212ms. G1 Eden Space: 763363328 -> 0; G1 Old Gen: 4215275512 -> 4466933752; G1 Su
[jira] [Commented] (CASSANDRA-14495) Memory Leak /High Memory usage post 3.11.2 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16640770#comment-16640770 ] Abdul Patel commented on CASSANDRA-14495: - Also is nodetool info best place to cheq heap usage ? > Memory Leak /High Memory usage post 3.11.2 upgrade > -- > > Key: CASSANDRA-14495 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14495 > Project: Cassandra > Issue Type: Bug > Components: Metrics >Reporter: Abdul Patel >Priority: Major > Attachments: cas_heap.txt > > > Hi All, > > I recently upgraded my non prod cassandra cluster( 4 nodes single DC) from > 3.10 to 3.11.2 version. > No issues reported apart from only nodetool info reporting 80% usage . > I intially had 16GB memory on each node, later i bumped up to 20GB, and > rebooted all nodes. > Waited for an week and now again i have seen memory usage more than 80% , > 16GB + . > this means some memory leaks are happening over the time. > Any one has faced such issue or do we have any workaround ? my 3.11.2 version > upgrade rollout has been halted because of this bug. > === > ID : 65b64f5a-7fe6-4036-94c8-8da9c57718cc > Gossip active : true > Thrift active : true > Native Transport active: true > Load : 985.24 MiB > Generation No : 1526923117 > Uptime (seconds) : 1097684 > Heap Memory (MB) : 16875.64 / 20480.00 > Off Heap Memory (MB) : 20.42 > Data Center : DC7 > Rack : rac1 > Exceptions : 0 > Key Cache : entries 3569, size 421.44 KiB, capacity 100 MiB, > 7931933 hits, 8098632 requests, 0.979 recent hit rate, 14400 save period in > seconds > Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 > requests, NaN recent hit rate, 0 save period in seconds > Counter Cache : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 > requests, NaN recent hit rate, 7200 save period in seconds > Chunk Cache : entries 2361, size 147.56 MiB, capacity 3.97 GiB, > 2412803 misses, 72594047 requests, 0.967 recent hit rate, NaN microseconds > miss latency > Percent Repaired : 99.88086234106282% > Token : (invoke with -T/--tokens to see all 256 tokens) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14495) Memory Leak /High Memory usage post 3.11.2 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16640766#comment-16640766 ] Abdul Patel commented on CASSANDRA-14495: - How frequently you had full gc job? Nodedool garbagecollect right? > Memory Leak /High Memory usage post 3.11.2 upgrade > -- > > Key: CASSANDRA-14495 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14495 > Project: Cassandra > Issue Type: Bug > Components: Metrics >Reporter: Abdul Patel >Priority: Major > Attachments: cas_heap.txt > > > Hi All, > > I recently upgraded my non prod cassandra cluster( 4 nodes single DC) from > 3.10 to 3.11.2 version. > No issues reported apart from only nodetool info reporting 80% usage . > I intially had 16GB memory on each node, later i bumped up to 20GB, and > rebooted all nodes. > Waited for an week and now again i have seen memory usage more than 80% , > 16GB + . > this means some memory leaks are happening over the time. > Any one has faced such issue or do we have any workaround ? my 3.11.2 version > upgrade rollout has been halted because of this bug. > === > ID : 65b64f5a-7fe6-4036-94c8-8da9c57718cc > Gossip active : true > Thrift active : true > Native Transport active: true > Load : 985.24 MiB > Generation No : 1526923117 > Uptime (seconds) : 1097684 > Heap Memory (MB) : 16875.64 / 20480.00 > Off Heap Memory (MB) : 20.42 > Data Center : DC7 > Rack : rac1 > Exceptions : 0 > Key Cache : entries 3569, size 421.44 KiB, capacity 100 MiB, > 7931933 hits, 8098632 requests, 0.979 recent hit rate, 14400 save period in > seconds > Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 > requests, NaN recent hit rate, 0 save period in seconds > Counter Cache : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 > requests, NaN recent hit rate, 7200 save period in seconds > Chunk Cache : entries 2361, size 147.56 MiB, capacity 3.97 GiB, > 2412803 misses, 72594047 requests, 0.967 recent hit rate, NaN microseconds > miss latency > Percent Repaired : 99.88086234106282% > Token : (invoke with -T/--tokens to see all 256 tokens) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14495) Memory Leak /High Memory usage post 3.11.2 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16640541#comment-16640541 ] Abdul Patel commented on CASSANDRA-14495: - i have seen same pattern in 3.11.3 as well , it works for 2-3 weeks and suddenly heap memory usage bumps up. and then every hr i get alerts. the only new thing is , i am installing cassandra reaper as well with new patch, but even with repaer down , i see same behavior. do we just bump of max heap ? or is it bug > Memory Leak /High Memory usage post 3.11.2 upgrade > -- > > Key: CASSANDRA-14495 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14495 > Project: Cassandra > Issue Type: Bug > Components: Metrics >Reporter: Abdul Patel >Priority: Major > Attachments: cas_heap.txt > > > Hi All, > > I recently upgraded my non prod cassandra cluster( 4 nodes single DC) from > 3.10 to 3.11.2 version. > No issues reported apart from only nodetool info reporting 80% usage . > I intially had 16GB memory on each node, later i bumped up to 20GB, and > rebooted all nodes. > Waited for an week and now again i have seen memory usage more than 80% , > 16GB + . > this means some memory leaks are happening over the time. > Any one has faced such issue or do we have any workaround ? my 3.11.2 version > upgrade rollout has been halted because of this bug. > === > ID : 65b64f5a-7fe6-4036-94c8-8da9c57718cc > Gossip active : true > Thrift active : true > Native Transport active: true > Load : 985.24 MiB > Generation No : 1526923117 > Uptime (seconds) : 1097684 > Heap Memory (MB) : 16875.64 / 20480.00 > Off Heap Memory (MB) : 20.42 > Data Center : DC7 > Rack : rac1 > Exceptions : 0 > Key Cache : entries 3569, size 421.44 KiB, capacity 100 MiB, > 7931933 hits, 8098632 requests, 0.979 recent hit rate, 14400 save period in > seconds > Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 > requests, NaN recent hit rate, 0 save period in seconds > Counter Cache : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 > requests, NaN recent hit rate, 7200 save period in seconds > Chunk Cache : entries 2361, size 147.56 MiB, capacity 3.97 GiB, > 2412803 misses, 72594047 requests, 0.967 recent hit rate, NaN microseconds > miss latency > Percent Repaired : 99.88086234106282% > Token : (invoke with -T/--tokens to see all 256 tokens) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14495) Memory Leak /High Memory usage post 3.11.2 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16553017#comment-16553017 ] Abdul Patel commented on CASSANDRA-14495: - no Gc logs and have to reboot the cluster every 2 weeks, anyone else faced the issue ? is it better to wait for 3.11.3? > Memory Leak /High Memory usage post 3.11.2 upgrade > -- > > Key: CASSANDRA-14495 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14495 > Project: Cassandra > Issue Type: Bug > Components: Metrics >Reporter: Abdul Patel >Priority: Major > Attachments: cas_heap.txt > > > Hi All, > > I recently upgraded my non prod cassandra cluster( 4 nodes single DC) from > 3.10 to 3.11.2 version. > No issues reported apart from only nodetool info reporting 80% usage . > I intially had 16GB memory on each node, later i bumped up to 20GB, and > rebooted all nodes. > Waited for an week and now again i have seen memory usage more than 80% , > 16GB + . > this means some memory leaks are happening over the time. > Any one has faced such issue or do we have any workaround ? my 3.11.2 version > upgrade rollout has been halted because of this bug. > === > ID : 65b64f5a-7fe6-4036-94c8-8da9c57718cc > Gossip active : true > Thrift active : true > Native Transport active: true > Load : 985.24 MiB > Generation No : 1526923117 > Uptime (seconds) : 1097684 > Heap Memory (MB) : 16875.64 / 20480.00 > Off Heap Memory (MB) : 20.42 > Data Center : DC7 > Rack : rac1 > Exceptions : 0 > Key Cache : entries 3569, size 421.44 KiB, capacity 100 MiB, > 7931933 hits, 8098632 requests, 0.979 recent hit rate, 14400 save period in > seconds > Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 > requests, NaN recent hit rate, 0 save period in seconds > Counter Cache : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 > requests, NaN recent hit rate, 7200 save period in seconds > Chunk Cache : entries 2361, size 147.56 MiB, capacity 3.97 GiB, > 2412803 misses, 72594047 requests, 0.967 recent hit rate, NaN microseconds > miss latency > Percent Repaired : 99.88086234106282% > Token : (invoke with -T/--tokens to see all 256 tokens) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14495) Memory Leak /High Memory usage post 3.11.2 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16517325#comment-16517325 ] Abdul Patel commented on CASSANDRA-14495: - i have gc log , it doesnt show much of full messages, could you guide on whats best way to check if GC or long pauses are reported? > Memory Leak /High Memory usage post 3.11.2 upgrade > -- > > Key: CASSANDRA-14495 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14495 > Project: Cassandra > Issue Type: Bug > Components: Metrics >Reporter: Abdul Patel >Priority: Major > Attachments: cas_heap.txt > > > Hi All, > > I recently upgraded my non prod cassandra cluster( 4 nodes single DC) from > 3.10 to 3.11.2 version. > No issues reported apart from only nodetool info reporting 80% usage . > I intially had 16GB memory on each node, later i bumped up to 20GB, and > rebooted all nodes. > Waited for an week and now again i have seen memory usage more than 80% , > 16GB + . > this means some memory leaks are happening over the time. > Any one has faced such issue or do we have any workaround ? my 3.11.2 version > upgrade rollout has been halted because of this bug. > === > ID : 65b64f5a-7fe6-4036-94c8-8da9c57718cc > Gossip active : true > Thrift active : true > Native Transport active: true > Load : 985.24 MiB > Generation No : 1526923117 > Uptime (seconds) : 1097684 > Heap Memory (MB) : 16875.64 / 20480.00 > Off Heap Memory (MB) : 20.42 > Data Center : DC7 > Rack : rac1 > Exceptions : 0 > Key Cache : entries 3569, size 421.44 KiB, capacity 100 MiB, > 7931933 hits, 8098632 requests, 0.979 recent hit rate, 14400 save period in > seconds > Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 > requests, NaN recent hit rate, 0 save period in seconds > Counter Cache : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 > requests, NaN recent hit rate, 7200 save period in seconds > Chunk Cache : entries 2361, size 147.56 MiB, capacity 3.97 GiB, > 2412803 misses, 72594047 requests, 0.967 recent hit rate, NaN microseconds > miss latency > Percent Repaired : 99.88086234106282% > Token : (invoke with -T/--tokens to see all 256 tokens) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14495) Memory Leak /High Memory usage post 3.11.2 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16517313#comment-16517313 ] Abdul Patel commented on CASSANDRA-14495: - We still got 80% and abover memory usage alert after a weeks time, whats the next stable version post 3.11.2? > Memory Leak /High Memory usage post 3.11.2 upgrade > -- > > Key: CASSANDRA-14495 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14495 > Project: Cassandra > Issue Type: Bug > Components: Metrics >Reporter: Abdul Patel >Priority: Major > Attachments: cas_heap.txt > > > Hi All, > > I recently upgraded my non prod cassandra cluster( 4 nodes single DC) from > 3.10 to 3.11.2 version. > No issues reported apart from only nodetool info reporting 80% usage . > I intially had 16GB memory on each node, later i bumped up to 20GB, and > rebooted all nodes. > Waited for an week and now again i have seen memory usage more than 80% , > 16GB + . > this means some memory leaks are happening over the time. > Any one has faced such issue or do we have any workaround ? my 3.11.2 version > upgrade rollout has been halted because of this bug. > === > ID : 65b64f5a-7fe6-4036-94c8-8da9c57718cc > Gossip active : true > Thrift active : true > Native Transport active: true > Load : 985.24 MiB > Generation No : 1526923117 > Uptime (seconds) : 1097684 > Heap Memory (MB) : 16875.64 / 20480.00 > Off Heap Memory (MB) : 20.42 > Data Center : DC7 > Rack : rac1 > Exceptions : 0 > Key Cache : entries 3569, size 421.44 KiB, capacity 100 MiB, > 7931933 hits, 8098632 requests, 0.979 recent hit rate, 14400 save period in > seconds > Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 > requests, NaN recent hit rate, 0 save period in seconds > Counter Cache : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 > requests, NaN recent hit rate, 7200 save period in seconds > Chunk Cache : entries 2361, size 147.56 MiB, capacity 3.97 GiB, > 2412803 misses, 72594047 requests, 0.967 recent hit rate, NaN microseconds > miss latency > Percent Repaired : 99.88086234106282% > Token : (invoke with -T/--tokens to see all 256 tokens) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14495) Memory Leak /High Memory usage post 3.11.2 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16503646#comment-16503646 ] Abdul Patel commented on CASSANDRA-14495: - I have lowered this to InitiatingHeapOccupancyPercent 30 for now, as default is 45%. I havent seen any memory consumption so far, but its been 3 days now, will monitor for a 8 to 10 days and see if it pops back in. I do agree apart from memory consumption no performance degradation or user compalints have been recieved so faar. > Memory Leak /High Memory usage post 3.11.2 upgrade > -- > > Key: CASSANDRA-14495 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14495 > Project: Cassandra > Issue Type: Bug > Components: Metrics >Reporter: Abdul Patel >Priority: Major > Attachments: cas_heap.txt > > > Hi All, > > I recently upgraded my non prod cassandra cluster( 4 nodes single DC) from > 3.10 to 3.11.2 version. > No issues reported apart from only nodetool info reporting 80% usage . > I intially had 16GB memory on each node, later i bumped up to 20GB, and > rebooted all nodes. > Waited for an week and now again i have seen memory usage more than 80% , > 16GB + . > this means some memory leaks are happening over the time. > Any one has faced such issue or do we have any workaround ? my 3.11.2 version > upgrade rollout has been halted because of this bug. > === > ID : 65b64f5a-7fe6-4036-94c8-8da9c57718cc > Gossip active : true > Thrift active : true > Native Transport active: true > Load : 985.24 MiB > Generation No : 1526923117 > Uptime (seconds) : 1097684 > Heap Memory (MB) : 16875.64 / 20480.00 > Off Heap Memory (MB) : 20.42 > Data Center : DC7 > Rack : rac1 > Exceptions : 0 > Key Cache : entries 3569, size 421.44 KiB, capacity 100 MiB, > 7931933 hits, 8098632 requests, 0.979 recent hit rate, 14400 save period in > seconds > Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 > requests, NaN recent hit rate, 0 save period in seconds > Counter Cache : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 > requests, NaN recent hit rate, 7200 save period in seconds > Chunk Cache : entries 2361, size 147.56 MiB, capacity 3.97 GiB, > 2412803 misses, 72594047 requests, 0.967 recent hit rate, NaN microseconds > miss latency > Percent Repaired : 99.88086234106282% > Token : (invoke with -T/--tokens to see all 256 tokens) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14495) Memory Leak /High Memory usage post 3.11.2 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16500541#comment-16500541 ] Abdul Patel commented on CASSANDRA-14495: - Its says below for G1GC: ## Optional G1 Settings # Save CPU time on large (>= 16GB) heaps by delaying region scanning # until the heap is 70% full. The default in Hotspot 8u40 is 40%. #-XX:InitiatingHeapOccupancyPercent=70 > Memory Leak /High Memory usage post 3.11.2 upgrade > -- > > Key: CASSANDRA-14495 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14495 > Project: Cassandra > Issue Type: Bug > Components: Metrics >Reporter: Abdul Patel >Priority: Major > Attachments: cas_heap.txt > > > Hi All, > > I recently upgraded my non prod cassandra cluster( 4 nodes single DC) from > 3.10 to 3.11.2 version. > No issues reported apart from only nodetool info reporting 80% usage . > I intially had 16GB memory on each node, later i bumped up to 20GB, and > rebooted all nodes. > Waited for an week and now again i have seen memory usage more than 80% , > 16GB + . > this means some memory leaks are happening over the time. > Any one has faced such issue or do we have any workaround ? my 3.11.2 version > upgrade rollout has been halted because of this bug. > === > ID : 65b64f5a-7fe6-4036-94c8-8da9c57718cc > Gossip active : true > Thrift active : true > Native Transport active: true > Load : 985.24 MiB > Generation No : 1526923117 > Uptime (seconds) : 1097684 > Heap Memory (MB) : 16875.64 / 20480.00 > Off Heap Memory (MB) : 20.42 > Data Center : DC7 > Rack : rac1 > Exceptions : 0 > Key Cache : entries 3569, size 421.44 KiB, capacity 100 MiB, > 7931933 hits, 8098632 requests, 0.979 recent hit rate, 14400 save period in > seconds > Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 > requests, NaN recent hit rate, 0 save period in seconds > Counter Cache : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 > requests, NaN recent hit rate, 7200 save period in seconds > Chunk Cache : entries 2361, size 147.56 MiB, capacity 3.97 GiB, > 2412803 misses, 72594047 requests, 0.967 recent hit rate, NaN microseconds > miss latency > Percent Repaired : 99.88086234106282% > Token : (invoke with -T/--tokens to see all 256 tokens) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14495) Memory Leak /High Memory usage post 3.11.2 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16500403#comment-16500403 ] Abdul Patel commented on CASSANDRA-14495: - got it , you are recommeding this parameter to be set @ 55? what would be downside of it? -XX:InitiatingHeapOccupancyPercent=70 > Memory Leak /High Memory usage post 3.11.2 upgrade > -- > > Key: CASSANDRA-14495 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14495 > Project: Cassandra > Issue Type: Bug > Components: Metrics >Reporter: Abdul Patel >Priority: Major > Attachments: cas_heap.txt > > > Hi All, > > I recently upgraded my non prod cassandra cluster( 4 nodes single DC) from > 3.10 to 3.11.2 version. > No issues reported apart from only nodetool info reporting 80% usage . > I intially had 16GB memory on each node, later i bumped up to 20GB, and > rebooted all nodes. > Waited for an week and now again i have seen memory usage more than 80% , > 16GB + . > this means some memory leaks are happening over the time. > Any one has faced such issue or do we have any workaround ? my 3.11.2 version > upgrade rollout has been halted because of this bug. > === > ID : 65b64f5a-7fe6-4036-94c8-8da9c57718cc > Gossip active : true > Thrift active : true > Native Transport active: true > Load : 985.24 MiB > Generation No : 1526923117 > Uptime (seconds) : 1097684 > Heap Memory (MB) : 16875.64 / 20480.00 > Off Heap Memory (MB) : 20.42 > Data Center : DC7 > Rack : rac1 > Exceptions : 0 > Key Cache : entries 3569, size 421.44 KiB, capacity 100 MiB, > 7931933 hits, 8098632 requests, 0.979 recent hit rate, 14400 save period in > seconds > Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 > requests, NaN recent hit rate, 0 save period in seconds > Counter Cache : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 > requests, NaN recent hit rate, 7200 save period in seconds > Chunk Cache : entries 2361, size 147.56 MiB, capacity 3.97 GiB, > 2412803 misses, 72594047 requests, 0.967 recent hit rate, NaN microseconds > miss latency > Percent Repaired : 99.88086234106282% > Token : (invoke with -T/--tokens to see all 256 tokens) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14495) Memory Leak /High Memory usage post 3.11.2 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16500389#comment-16500389 ] Abdul Patel commented on CASSANDRA-14495: - I am more concern on why this sudden behaviour in this version only ? could you please share inputs on how i can work on =decrease initiating occupancy (55% I'd recommend) ? we already have G1GC setup, anything else can be done > Memory Leak /High Memory usage post 3.11.2 upgrade > -- > > Key: CASSANDRA-14495 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14495 > Project: Cassandra > Issue Type: Bug > Components: Metrics >Reporter: Abdul Patel >Priority: Major > Attachments: cas_heap.txt > > > Hi All, > > I recently upgraded my non prod cassandra cluster( 4 nodes single DC) from > 3.10 to 3.11.2 version. > No issues reported apart from only nodetool info reporting 80% usage . > I intially had 16GB memory on each node, later i bumped up to 20GB, and > rebooted all nodes. > Waited for an week and now again i have seen memory usage more than 80% , > 16GB + . > this means some memory leaks are happening over the time. > Any one has faced such issue or do we have any workaround ? my 3.11.2 version > upgrade rollout has been halted because of this bug. > === > ID : 65b64f5a-7fe6-4036-94c8-8da9c57718cc > Gossip active : true > Thrift active : true > Native Transport active: true > Load : 985.24 MiB > Generation No : 1526923117 > Uptime (seconds) : 1097684 > Heap Memory (MB) : 16875.64 / 20480.00 > Off Heap Memory (MB) : 20.42 > Data Center : DC7 > Rack : rac1 > Exceptions : 0 > Key Cache : entries 3569, size 421.44 KiB, capacity 100 MiB, > 7931933 hits, 8098632 requests, 0.979 recent hit rate, 14400 save period in > seconds > Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 > requests, NaN recent hit rate, 0 save period in seconds > Counter Cache : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 > requests, NaN recent hit rate, 7200 save period in seconds > Chunk Cache : entries 2361, size 147.56 MiB, capacity 3.97 GiB, > 2412803 misses, 72594047 requests, 0.967 recent hit rate, NaN microseconds > miss latency > Percent Repaired : 99.88086234106282% > Token : (invoke with -T/--tokens to see all 256 tokens) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14495) Memory Leak /High Memory usage post 3.11.2 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16500366#comment-16500366 ] Abdul Patel commented on CASSANDRA-14495: - Attached the output , out of 20GB 14Gb is used on one of node for now > Memory Leak /High Memory usage post 3.11.2 upgrade > -- > > Key: CASSANDRA-14495 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14495 > Project: Cassandra > Issue Type: Bug > Components: Metrics >Reporter: Abdul Patel >Priority: Major > Attachments: cas_heap.txt > > > Hi All, > > I recently upgraded my non prod cassandra cluster( 4 nodes single DC) from > 3.10 to 3.11.2 version. > No issues reported apart from only nodetool info reporting 80% usage . > I intially had 16GB memory on each node, later i bumped up to 20GB, and > rebooted all nodes. > Waited for an week and now again i have seen memory usage more than 80% , > 16GB + . > this means some memory leaks are happening over the time. > Any one has faced such issue or do we have any workaround ? my 3.11.2 version > upgrade rollout has been halted because of this bug. > === > ID : 65b64f5a-7fe6-4036-94c8-8da9c57718cc > Gossip active : true > Thrift active : true > Native Transport active: true > Load : 985.24 MiB > Generation No : 1526923117 > Uptime (seconds) : 1097684 > Heap Memory (MB) : 16875.64 / 20480.00 > Off Heap Memory (MB) : 20.42 > Data Center : DC7 > Rack : rac1 > Exceptions : 0 > Key Cache : entries 3569, size 421.44 KiB, capacity 100 MiB, > 7931933 hits, 8098632 requests, 0.979 recent hit rate, 14400 save period in > seconds > Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 > requests, NaN recent hit rate, 0 save period in seconds > Counter Cache : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 > requests, NaN recent hit rate, 7200 save period in seconds > Chunk Cache : entries 2361, size 147.56 MiB, capacity 3.97 GiB, > 2412803 misses, 72594047 requests, 0.967 recent hit rate, NaN microseconds > miss latency > Percent Repaired : 99.88086234106282% > Token : (invoke with -T/--tokens to see all 256 tokens) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14495) Memory Leak /High Memory usage post 3.11.2 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abdul Patel updated CASSANDRA-14495: Attachment: cas_heap.txt > Memory Leak /High Memory usage post 3.11.2 upgrade > -- > > Key: CASSANDRA-14495 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14495 > Project: Cassandra > Issue Type: Bug > Components: Metrics >Reporter: Abdul Patel >Priority: Major > Attachments: cas_heap.txt > > > Hi All, > > I recently upgraded my non prod cassandra cluster( 4 nodes single DC) from > 3.10 to 3.11.2 version. > No issues reported apart from only nodetool info reporting 80% usage . > I intially had 16GB memory on each node, later i bumped up to 20GB, and > rebooted all nodes. > Waited for an week and now again i have seen memory usage more than 80% , > 16GB + . > this means some memory leaks are happening over the time. > Any one has faced such issue or do we have any workaround ? my 3.11.2 version > upgrade rollout has been halted because of this bug. > === > ID : 65b64f5a-7fe6-4036-94c8-8da9c57718cc > Gossip active : true > Thrift active : true > Native Transport active: true > Load : 985.24 MiB > Generation No : 1526923117 > Uptime (seconds) : 1097684 > Heap Memory (MB) : 16875.64 / 20480.00 > Off Heap Memory (MB) : 20.42 > Data Center : DC7 > Rack : rac1 > Exceptions : 0 > Key Cache : entries 3569, size 421.44 KiB, capacity 100 MiB, > 7931933 hits, 8098632 requests, 0.979 recent hit rate, 14400 save period in > seconds > Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 > requests, NaN recent hit rate, 0 save period in seconds > Counter Cache : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 > requests, NaN recent hit rate, 7200 save period in seconds > Chunk Cache : entries 2361, size 147.56 MiB, capacity 3.97 GiB, > 2412803 misses, 72594047 requests, 0.967 recent hit rate, NaN microseconds > miss latency > Percent Repaired : 99.88086234106282% > Token : (invoke with -T/--tokens to see all 256 tokens) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16500249#comment-16500249
]
Abdul Patel commented on CASSANDRA-14355:
-
If 3.11.2 release has memory leak issues, whats the next best patch version ?
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Major
> Fix For: 3.11.3
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14495) Memory Leak /High Memory usage post 3.11.2 upgrade
Abdul Patel created CASSANDRA-14495: --- Summary: Memory Leak /High Memory usage post 3.11.2 upgrade Key: CASSANDRA-14495 URL: https://issues.apache.org/jira/browse/CASSANDRA-14495 Project: Cassandra Issue Type: Bug Components: Metrics Reporter: Abdul Patel Hi All, I recently upgraded my non prod cassandra cluster( 4 nodes single DC) from 3.10 to 3.11.2 version. No issues reported apart from only nodetool info reporting 80% usage . I intially had 16GB memory on each node, later i bumped up to 20GB, and rebooted all nodes. Waited for an week and now again i have seen memory usage more than 80% , 16GB + . this means some memory leaks are happening over the time. Any one has faced such issue or do we have any workaround ? my 3.11.2 version upgrade rollout has been halted because of this bug. === ID : 65b64f5a-7fe6-4036-94c8-8da9c57718cc Gossip active : true Thrift active : true Native Transport active: true Load : 985.24 MiB Generation No : 1526923117 Uptime (seconds) : 1097684 Heap Memory (MB) : 16875.64 / 20480.00 Off Heap Memory (MB) : 20.42 Data Center : DC7 Rack : rac1 Exceptions : 0 Key Cache : entries 3569, size 421.44 KiB, capacity 100 MiB, 7931933 hits, 8098632 requests, 0.979 recent hit rate, 14400 save period in seconds Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds Counter Cache : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds Chunk Cache : entries 2361, size 147.56 MiB, capacity 3.97 GiB, 2412803 misses, 72594047 requests, 0.967 recent hit rate, NaN microseconds miss latency Percent Repaired : 99.88086234106282% Token : (invoke with -T/--tokens to see all 256 tokens) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16499444#comment-16499444
]
Abdul Patel commented on CASSANDRA-14355:
-
I have seen nodetool info reporting high memory usage after a weeks of
upgrading t0 3.11.2, i rebooted all nodes 1 week no issues, later agian same ,
no errors in errorlog .
is this related to this issue?
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Major
> Fix For: 3.11.3
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14451) Infinity ms Commit Log Sync
[
https://issues.apache.org/jira/browse/CASSANDRA-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16499232#comment-16499232
]
Abdul Patel commented on CASSANDRA-14451:
-
Can this cause Memory usage to go high ?
after 3.11.2 upgrade i am getting high memory usage alerts and nothing in
errorlog sometime , but sometime i see same PERIODIC-COMMIT-LOG-SYNCER message
> Infinity ms Commit Log Sync
> ---
>
> Key: CASSANDRA-14451
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14451
> Project: Cassandra
> Issue Type: Bug
> Environment: 3.11.2 - 2 DC
>Reporter: Harry Hough
>Assignee: Jason Brown
>Priority: Minor
> Fix For: 3.0.x, 3.11.x, 4.0.x
>
>
> Its giving commit log sync warnings where there were apparently zero syncs
> and therefore gives "Infinityms" as the average duration
> {code:java}
> WARN [PERIODIC-COMMIT-LOG-SYNCER] 2018-05-16 21:11:14,294
> NoSpamLogger.java:94 - Out of 0 commit log syncs over the past 0.00s with
> average duration of Infinityms, 1 have exceeded the configured commit
> interval by an average of 74.40ms
> WARN [PERIODIC-COMMIT-LOG-SYNCER] 2018-05-16 21:16:57,844
> NoSpamLogger.java:94 - Out of 0 commit log syncs over the past 0.00s with
> average duration of Infinityms, 1 have exceeded the configured commit
> interval by an average of 198.69ms
> WARN [PERIODIC-COMMIT-LOG-SYNCER] 2018-05-16 21:24:46,325
> NoSpamLogger.java:94 - Out of 0 commit log syncs over the past 0.00s with
> average duration of Infinityms, 1 have exceeded the configured commit
> interval by an average of 264.11ms
> WARN [PERIODIC-COMMIT-LOG-SYNCER] 2018-05-16 21:29:46,393
> NoSpamLogger.java:94 - Out of 32 commit log syncs over the past 268.84s with,
> average duration of 17.56ms, 1 have exceeded the configured commit interval
> by an average of 173.66ms{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14451) Infinity ms Commit Log Sync
[
https://issues.apache.org/jira/browse/CASSANDRA-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16484164#comment-16484164
]
Abdul Patel commented on CASSANDRA-14451:
-
all,
even i recently upgraded from 3.10 to 3.11.2 and recieving similar errros
PERIODIC-COMMIT-LOG-SYNCER.
One thing i observed is even cqlsh login takes over a minute when these erros
are reported., also my lowest non prod with 4 nodes 1 DC setup is showing this
error and memory was configured as 16GB, the usage went on to 13 to 14 GB.
seems to be related .
Is this bug with 3.11.2 ?
> Infinity ms Commit Log Sync
> ---
>
> Key: CASSANDRA-14451
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14451
> Project: Cassandra
> Issue Type: Bug
> Environment: 3.11.2 - 2 DC
>Reporter: Harry Hough
>Assignee: Jason Brown
>Priority: Minor
>
> Its giving commit log sync warnings where there were apparently zero syncs
> and therefore gives "Infinityms" as the average duration
> {code:java}
> WARN [PERIODIC-COMMIT-LOG-SYNCER] 2018-05-16 21:11:14,294
> NoSpamLogger.java:94 - Out of 0 commit log syncs over the past 0.00s with
> average duration of Infinityms, 1 have exceeded the configured commit
> interval by an average of 74.40ms
> WARN [PERIODIC-COMMIT-LOG-SYNCER] 2018-05-16 21:16:57,844
> NoSpamLogger.java:94 - Out of 0 commit log syncs over the past 0.00s with
> average duration of Infinityms, 1 have exceeded the configured commit
> interval by an average of 198.69ms
> WARN [PERIODIC-COMMIT-LOG-SYNCER] 2018-05-16 21:24:46,325
> NoSpamLogger.java:94 - Out of 0 commit log syncs over the past 0.00s with
> average duration of Infinityms, 1 have exceeded the configured commit
> interval by an average of 264.11ms
> WARN [PERIODIC-COMMIT-LOG-SYNCER] 2018-05-16 21:29:46,393
> NoSpamLogger.java:94 - Out of 32 commit log syncs over the past 268.84s with,
> average duration of 17.56ms, 1 have exceeded the configured commit interval
> by an average of 173.66ms{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
