[jira] [Commented] (CASSANDRA-15397) IntervalTree performance comparison with Linear Walk and Binary Search based Elimination.
[ https://issues.apache.org/jira/browse/CASSANDRA-15397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993222#comment-16993222 ] Chandrasekhar Thumuluru commented on CASSANDRA-15397: - [~benedict] — When you get a chance, can you review the changes and let me know your feedback? > IntervalTree performance comparison with Linear Walk and Binary Search based > Elimination. > -- > > Key: CASSANDRA-15397 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15397 > Project: Cassandra > Issue Type: Improvement > Components: Local/SSTable >Reporter: Chandrasekhar Thumuluru >Assignee: Chandrasekhar Thumuluru >Priority: Low > Attachments: 95p_1_SSTable_with_5000_Searches.png, > 95p_15000_SSTable_with_5000_Searches.png, > 95p_2_SSTable_with_5000_Searches.png, > 95p_25000_SSTable_with_5000_Searches.png, > 95p_3_SSTable_with_5000_Searches.png, > 95p_5000_SSTable_with_5000_Searches.png, > 99p_1_SSTable_with_5000_Searches.png, > 99p_15000_SSTable_with_5000_Searches.png, > 99p_2_SSTable_with_5000_Searches.png, > 99p_25000_SSTable_with_5000_Searches.png, > 99p_3_SSTable_with_5000_Searches.png, > 99p_5000_SSTable_with_5000_Searches.png, IntervalList.java, > IntervalListWithElimination.java, IntervalTreeSimplified.java, > Mean_1_SSTable_with_5000_Searches.png, > Mean_15000_SSTable_with_5000_Searches.png, > Mean_2_SSTable_with_5000_Searches.png, > Mean_25000_SSTable_with_5000_Searches.png, > Mean_3_SSTable_with_5000_Searches.png, > Mean_5000_SSTable_with_5000_Searches.png, TESTS-TestSuites.xml.lz4, > replace_intervaltree_with_intervallist.patch > > > Cassandra uses IntervalTrees to identify the SSTables that overlap with > search interval. In Cassandra, IntervalTrees are not mutated. They are > recreated each time a mutation is required. This can be an issue during > repairs. In fact we noticed such issues during repair. > Since lists are cache friendly compared to linked lists and trees, I decided > to compare the search performance with: > * Linear Walk. > * Elimination using Binary Search (idea is to eliminate intervals using start > and end points of search interval). > Based on the tests I ran, I noticed Binary Search based elimination almost > always performs similar to IntervalTree or out performs IntervalTree based > search. The cost of IntervalTree construction is also substantial and > produces lot of garbage during repairs. > I ran the tests using random intervals to build the tree/lists and another > randomly generated search interval with 5000 iterations. I'm attaching all > the relevant graphs. The x-axis in the graphs is the search interval > coverage. 10p means the search interval covered 10% of the intervals. The > y-axis is the time the search took in nanos. > PS: > # For the purpose of test, I simplified the IntervalTree by removing the data > portion of the interval. Modified the template version (Java generics) to a > specialized version. > # I used the code from Cassandra version _3.11_. > # Time in the graph is in nanos. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15434) Make test module for nodetools
[ https://issues.apache.org/jira/browse/CASSANDRA-15434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993192#comment-16993192 ] maxwellguo commented on CASSANDRA-15434: yes,After 15429 RESOLVED, I will try to make some test for nodetool .:) > Make test module for nodetools > --- > > Key: CASSANDRA-15434 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15434 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest, Test/unit, Tool/nodetool >Reporter: maxwellguo >Assignee: maxwellguo >Priority: Low > > All nodetools feature don't got a unit/dtest module. We should add one for it -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15388) Add compaction allocation measurement test to support compaction gc optimization.
[
https://issues.apache.org/jira/browse/CASSANDRA-15388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993136#comment-16993136
]
Benedict Elliott Smith commented on CASSANDRA-15388:
With absolutely no preconceived answer to this question on my part: have you
considered and rejected JMH's GC profiler?
I'm unsure how the approaches compare with respect to sophistication (I haven't
looked closely at how either underlying tool is implemented), but having played
with it myself recently, using jmh 1.22, it seems to produce exactly the
numbers I would calculate manually for small functions performing easily
calculated allocations.
I only ask because it would hook into our (minimal) pre-existing benchmark
workflows, and produce throughput/latency data alongside. The latest version
of jmh can even neatly hook into perf counters to produce very sophisticated
ancillary data to consider at the same time, to weigh the pros/cons of a change
(although it's definitely not as easy to explore as a flamegraph).
> Add compaction allocation measurement test to support compaction gc
> optimization.
> --
>
> Key: CASSANDRA-15388
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15388
> Project: Cassandra
> Issue Type: Sub-task
> Components: Local/Compaction
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 4.0
>
>
> This adds a test that is able to quickly and accurately measure the effect of
> potential gc optimizations against a wide range of (synthetic) compaction
> workloads. This test accurately measures allocation rates from 16 workloads
> in less that 2 minutes.
> This test uses google’s {{java-allocation-instrumenter}} agent to measure the
> workloads. Measurements using this agent are very accurate and pretty
> repeatable from run to run, with most variance being negligible (1-2 bytes
> per partition), although workloads with larger but fewer partitions vary a
> bit more (still less that 0.03%).
> The thinking behind this patch is that with compaction, we’re generally
> interested in the memory allocated per partition, since garbage scales more
> or less linearly with the number of partitions compacted. So measuring
> allocation from a small number of partitions that otherwise represent real
> world use cases is a good enough approximation.
> In addition to helping with compaction optimizations, this test could be used
> as a template for future optimization work. This pattern could also be used
> to set allocation limits on workloads/operations and fail CI if the
> allocation behavior changes past some threshold.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15429) Support NodeTool for in-jvm dtest
[ https://issues.apache.org/jira/browse/CASSANDRA-15429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993122#comment-16993122 ] David Capwell commented on CASSANDRA-15429: --- LGTM +1 > Support NodeTool for in-jvm dtest > - > > Key: CASSANDRA-15429 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15429 > Project: Cassandra > Issue Type: New Feature > Components: Test/dtest >Reporter: Yifan Cai >Assignee: Yifan Cai >Priority: Normal > Labels: pull-request-available > Fix For: 4.0 > > Time Spent: 4.5h > Remaining Estimate: 0h > > In-JVM dtest framework does not support nodetool as of now. This > functionality is wanted in some tests, e.g. constructing an end-to-end test > scenario that uses nodetool. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15429) Support NodeTool for in-jvm dtest
[ https://issues.apache.org/jira/browse/CASSANDRA-15429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993080#comment-16993080 ] Yifan Cai commented on CASSANDRA-15429: --- Thanks [~dcapwell]! PRs have been updated to address the review comments. > Support NodeTool for in-jvm dtest > - > > Key: CASSANDRA-15429 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15429 > Project: Cassandra > Issue Type: New Feature > Components: Test/dtest >Reporter: Yifan Cai >Assignee: Yifan Cai >Priority: Normal > Labels: pull-request-available > Fix For: 4.0 > > Time Spent: 4h 20m > Remaining Estimate: 0h > > In-JVM dtest framework does not support nodetool as of now. This > functionality is wanted in some tests, e.g. constructing an end-to-end test > scenario that uses nodetool. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15429) Support NodeTool for in-jvm dtest
[ https://issues.apache.org/jira/browse/CASSANDRA-15429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993049#comment-16993049 ] David Capwell commented on CASSANDRA-15429: --- Did another pass. Mostly LGTM; left a few comments in GitHub. > Support NodeTool for in-jvm dtest > - > > Key: CASSANDRA-15429 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15429 > Project: Cassandra > Issue Type: New Feature > Components: Test/dtest >Reporter: Yifan Cai >Assignee: Yifan Cai >Priority: Normal > Labels: pull-request-available > Fix For: 4.0 > > Time Spent: 4h 10m > Remaining Estimate: 0h > > In-JVM dtest framework does not support nodetool as of now. This > functionality is wanted in some tests, e.g. constructing an end-to-end test > scenario that uses nodetool. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15393) Add byte array backed cells
[ https://issues.apache.org/jira/browse/CASSANDRA-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Blake Eggleston updated CASSANDRA-15393: Reviewers: Benedict Elliott Smith > Add byte array backed cells > --- > > Key: CASSANDRA-15393 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15393 > Project: Cassandra > Issue Type: Sub-task > Components: Local/Compaction >Reporter: Blake Eggleston >Assignee: Blake Eggleston >Priority: Normal > Fix For: 4.0 > > > We currently materialize all values as on heap byte buffers. Byte buffers > have a fairly high overhead given how frequently they’re used, and on the > compaction and local read path we don’t do anything that needs them. Use of > byte buffer methods only happens on the coordinator. Using cells that are > backed by byte arrays instead in these situations reduces compaction and read > garbage up to 22% in many cases. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15392) Pool Merge Iterators
[ https://issues.apache.org/jira/browse/CASSANDRA-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Blake Eggleston updated CASSANDRA-15392: Reviewers: Benedict Elliott Smith > Pool Merge Iterators > > > Key: CASSANDRA-15392 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15392 > Project: Cassandra > Issue Type: Sub-task > Components: Local/Compaction >Reporter: Blake Eggleston >Assignee: Blake Eggleston >Priority: Normal > Fix For: 4.0 > > > By pooling merge iterators, instead of creating new ones each time we need > them, we can reduce garbage on the compaction and read paths under relevant > workloads by ~4% in many cases. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15388) Add compaction allocation measurement test to support compaction gc optimization.
[
https://issues.apache.org/jira/browse/CASSANDRA-15388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Blake Eggleston updated CASSANDRA-15388:
Reviewers: Benedict Elliott Smith
> Add compaction allocation measurement test to support compaction gc
> optimization.
> --
>
> Key: CASSANDRA-15388
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15388
> Project: Cassandra
> Issue Type: Sub-task
> Components: Local/Compaction
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 4.0
>
>
> This adds a test that is able to quickly and accurately measure the effect of
> potential gc optimizations against a wide range of (synthetic) compaction
> workloads. This test accurately measures allocation rates from 16 workloads
> in less that 2 minutes.
> This test uses google’s {{java-allocation-instrumenter}} agent to measure the
> workloads. Measurements using this agent are very accurate and pretty
> repeatable from run to run, with most variance being negligible (1-2 bytes
> per partition), although workloads with larger but fewer partitions vary a
> bit more (still less that 0.03%).
> The thinking behind this patch is that with compaction, we’re generally
> interested in the memory allocated per partition, since garbage scales more
> or less linearly with the number of partitions compacted. So measuring
> allocation from a small number of partitions that otherwise represent real
> world use cases is a good enough approximation.
> In addition to helping with compaction optimizations, this test could be used
> as a template for future optimization work. This pattern could also be used
> to set allocation limits on workloads/operations and fail CI if the
> allocation behavior changes past some threshold.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15391) Reduce heap footprint of commonly allocated objects
[ https://issues.apache.org/jira/browse/CASSANDRA-15391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Blake Eggleston updated CASSANDRA-15391: Reviewers: Benedict Elliott Smith > Reduce heap footprint of commonly allocated objects > --- > > Key: CASSANDRA-15391 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15391 > Project: Cassandra > Issue Type: Sub-task > Components: Local/Compaction >Reporter: Blake Eggleston >Assignee: Blake Eggleston >Priority: Normal > Fix For: 4.0 > > > BufferCell, BTreeRow, and Clustering make up a significant amount of > allocations during reads/compactions, and many of the fields of these classes > are often unused. For example, the CellPath reference in BufferCell is only > every used for collection columns. Since we know which fields will and won’t > be used during cell creation, we can define specialized classes that only > take up heap space for the data they’ll be using. This reduces compaction > garbage by up to 4.5%. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15442) Read repair implicitly increases read timeout value
[
https://issues.apache.org/jira/browse/CASSANDRA-15442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993037#comment-16993037
]
Blake Eggleston commented on CASSANDRA-15442:
-
No argument that using the read timeout is more understandable from the client
perspective, my concern is unintended consequences of silently increasing the
write timeout. Granted I can't think of any off the top of my head.
[~clohfink], [~jjirsa], can you think of anything bad that might happen if we
did that?
> Read repair implicitly increases read timeout value
> ---
>
> Key: CASSANDRA-15442
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15442
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
>Reporter: Yifan Cai
>Assignee: Yifan Cai
>Priority: Normal
>
> When read repair occurs during a read, internally, it starts several
> _blocking_ operations in sequence. See
> {{org.apache.cassandra.service.StorageProxy#fetchRows}}.
> The timeline of the blocking operations
> # Regular read, wait for full data/digest read response to complete.
> {{reads[*].awaitResponses();}}
> # Read repair read, wait for full data read response to complete.
> {{reads[*].awaitReadRepair();}}
> # Read repair write, wait for write response to complete.
> {{concatAndBlockOnRepair(results, repairs);}}
> Step 1 and 2 share the same timeout, and wait for the duration of read
> timeout, say 5 s.
> Step 3 waits for the duration of write timeout, say 2 s.
> In the worse case, the actual time taken for a read could accumulate to ~7 s,
> if each individual step does not exceed the timeout value.
> From the client perspective, it may not expect a request taken higher than
> the database configured timeout value.
> Such scenario is especially bad for the clients that have set up client-side
> timeout monitoring close to the configured one. The clients think the
> operations timed out and abort, but they are in fact still running on server.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992980#comment-16992980
]
Benedict Elliott Smith commented on CASSANDRA-14355:
If there's an issue around Cassandra failing to clean up after a terminated
repair, that's certainly a bug you should file.
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Normal
> Fix For: 3.11.x
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png, LongGC_Dominator-Tree.png, LongGC_Histogram.png,
> LongGC_Problem-Suspect-1_FastThreadLocalThread.png, LongGC_nodetool_info.txt
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992973#comment-16992973
]
Chris Kistner commented on CASSANDRA-14355:
---
[~benedict], thank you for the feedback.
After we've done some extensive investigation on our end, we pinned our issue
to Cassandra Reaper terminating the Cassandra repair sessions if they don't
finish within 30 minutes ("hangingRepairTimeoutMins" default), and then those
repair threads aren't closed by Cassandra 3.11.4 and thus retain ~200MB of data
on heap in our case. The other annoying part when this happens is that both
Cassandra & Cassandra Reaper logs it as if the repair session was successful,
unless you go and look at the logs in detail.
I think I should rather create a new bug for Cassandra not closing the Repair
session thread correctly, since the original issue's
"io.netty.util.concurrent.FastThreadLocalThread" class was referenced from
"Native-Transport-Requests" and "ReadStage" threads, where as ours were all
from "Repair" threads.
We might be able to see if we can reproduce the Cassandra issue where it is not
closing Repair threads properly on 3.11.4 and then on 3.11.5.
So far we've only experienced these OOME issues in client production
environments and right now all our clients are in code/change freeze, so we
won't be able to test 3.11.5 for about a month in a large production
environment.
So for now we'll just schedule per host & per table repairs using cron scripts
and doing a different host each day of the week instead of using Cassandra
Reaper that might terminate our repair sessions.
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Normal
> Fix For: 3.11.x
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png, LongGC_Dominator-Tree.png, LongGC_Histogram.png,
> LongGC_Problem-Suspect-1_FastThreadLocalThread.png, LongGC_nodetool_info.txt
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15442) Read repair implicitly increases read timeout value
[
https://issues.apache.org/jira/browse/CASSANDRA-15442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992941#comment-16992941
]
Jordan West commented on CASSANDRA-15442:
-
Thanks for the patch [~yifanc]. I've actually wondered about it for a while but
never thought of it as a bug (assumed it was by design). Glad to see it
addressed. A few minor nits, all optional:
* I agree that we should only use the repair timeout for the reason you
mentioned. I find it confusing that the write timeout is involved since the
client has no idea if read repair will occur when submitting the read.
* Since `awaitRepairs` is only called from `awaitRepairsUntil` consider
merging the two functions. It might make the code more clear and require less
jumping around.
* I agree with [~bdeggleston] that setting the write timeout very high in your
test can show how the change ensures independence from it – regardless of any
other code changes. It may also give you a more flexible way to write the
assertion at the end of `readRepairTimeoutTest()`.
* Additionally, re: the two `Assert.assertTrue`s at the end of
`readRepairTimeoutTest`: consider adding failure messages to make them more
descriptive, as well as collapsing it in to one expression (e.g
`actualTimeTaken < reducedReadTimeout + magicDelayAmount && actualTimeTaken >
reducedReadTimeout`).
> Read repair implicitly increases read timeout value
> ---
>
> Key: CASSANDRA-15442
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15442
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
>Reporter: Yifan Cai
>Assignee: Yifan Cai
>Priority: Normal
>
> When read repair occurs during a read, internally, it starts several
> _blocking_ operations in sequence. See
> {{org.apache.cassandra.service.StorageProxy#fetchRows}}.
> The timeline of the blocking operations
> # Regular read, wait for full data/digest read response to complete.
> {{reads[*].awaitResponses();}}
> # Read repair read, wait for full data read response to complete.
> {{reads[*].awaitReadRepair();}}
> # Read repair write, wait for write response to complete.
> {{concatAndBlockOnRepair(results, repairs);}}
> Step 1 and 2 share the same timeout, and wait for the duration of read
> timeout, say 5 s.
> Step 3 waits for the duration of write timeout, say 2 s.
> In the worse case, the actual time taken for a read could accumulate to ~7 s,
> if each individual step does not exceed the timeout value.
> From the client perspective, it may not expect a request taken higher than
> the database configured timeout value.
> Such scenario is especially bad for the clients that have set up client-side
> timeout monitoring close to the configured one. The clients think the
> operations timed out and abort, but they are in fact still running on server.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15442) Read repair implicitly increases read timeout value
[
https://issues.apache.org/jira/browse/CASSANDRA-15442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West updated CASSANDRA-15442:
Reviewers: Blake Eggleston, Jordan West (was: Blake Eggleston)
> Read repair implicitly increases read timeout value
> ---
>
> Key: CASSANDRA-15442
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15442
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
>Reporter: Yifan Cai
>Assignee: Yifan Cai
>Priority: Normal
>
> When read repair occurs during a read, internally, it starts several
> _blocking_ operations in sequence. See
> {{org.apache.cassandra.service.StorageProxy#fetchRows}}.
> The timeline of the blocking operations
> # Regular read, wait for full data/digest read response to complete.
> {{reads[*].awaitResponses();}}
> # Read repair read, wait for full data read response to complete.
> {{reads[*].awaitReadRepair();}}
> # Read repair write, wait for write response to complete.
> {{concatAndBlockOnRepair(results, repairs);}}
> Step 1 and 2 share the same timeout, and wait for the duration of read
> timeout, say 5 s.
> Step 3 waits for the duration of write timeout, say 2 s.
> In the worse case, the actual time taken for a read could accumulate to ~7 s,
> if each individual step does not exceed the timeout value.
> From the client perspective, it may not expect a request taken higher than
> the database configured timeout value.
> Such scenario is especially bad for the clients that have set up client-side
> timeout monitoring close to the configured one. The clients think the
> operations timed out and abort, but they are in fact still running on server.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15429) Support NodeTool for in-jvm dtest
[ https://issues.apache.org/jira/browse/CASSANDRA-15429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992882#comment-16992882 ] David Capwell commented on CASSANDRA-15429: --- We should refactor NodeProbe and InternalNodeProbe so InternalNodeProbe just overrides how NodeProbe gets the mbeans; right now there is a lot of rewriting of methods which can go away by doing that. This is my main comment, I left a few minor ones in GitHub. > Support NodeTool for in-jvm dtest > - > > Key: CASSANDRA-15429 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15429 > Project: Cassandra > Issue Type: New Feature > Components: Test/dtest >Reporter: Yifan Cai >Assignee: Yifan Cai >Priority: Normal > Labels: pull-request-available > Fix For: 4.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > In-JVM dtest framework does not support nodetool as of now. This > functionality is wanted in some tests, e.g. constructing an end-to-end test > scenario that uses nodetool. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15429) Support NodeTool for in-jvm dtest
[
https://issues.apache.org/jira/browse/CASSANDRA-15429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992778#comment-16992778
]
Yifan Cai commented on CASSANDRA-15429:
---
Thanks [~ifesdjeen]
{quote}Can we hide nodetool behind a feature flag, same as native transport and
other things?
{quote}
There are two reasons that I think nodetool does not need a feature flag.
# {{nodetool}} only consumes resources when being invoked, and the resources
are released once the call exit. On instance startup, we only replace the
implementation of nodetool probe (to the internal one).
# A new method {{nodetool}} is added in {{IInstance}}, and use the
mock/internal node probe at the instance impl. If a feature flag is introduced,
the behavior when the nodetool feature is disabled and calling {{nodetool}}
method could be confusing. Because, the actual node probe cannot connect to the
JVM cluster instances.
{quote}a TODO about GCInspector, can you elaborate a bit?
{quote}
Nice catch! The TODO should be removed. The patch does not support the nodetool
commands that are used to print information to stdout, e.g. GCInspector, which
is used to print the gc stats. The text to stdout is less useful for jvm dtest.
I have removed the todo and rebased the branches onto the latest.
> Support NodeTool for in-jvm dtest
> -
>
> Key: CASSANDRA-15429
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15429
> Project: Cassandra
> Issue Type: New Feature
> Components: Test/dtest
>Reporter: Yifan Cai
>Assignee: Yifan Cai
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> In-JVM dtest framework does not support nodetool as of now. This
> functionality is wanted in some tests, e.g. constructing an end-to-end test
> scenario that uses nodetool.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15446) Per-thread stack size is too small on aarch64 CentOS
[ https://issues.apache.org/jira/browse/CASSANDRA-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992744#comment-16992744 ] David Capwell commented on CASSANDRA-15446: --- This looks related to https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8170655 and https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8173339; the reported JVM is 1.8, does 11 work? > Per-thread stack size is too small on aarch64 CentOS > > > Key: CASSANDRA-15446 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15446 > Project: Cassandra > Issue Type: Bug > Components: Local/Config >Reporter: Heming Fu >Assignee: Heming Fu >Priority: Normal > Fix For: 3.11.5, 2.1.x, 2.2.x, 3.0.x > > > Hi all, > I found an issue when I tried to start cassandra on my aarch64 CentOS7.6, > however no errors on Ubuntu. Of course I could increase -Xss in jvm.options > to fix it, but this issue also caused Cassandra's docker images from docker > hub could not run containers on this OS. > The information of my current environment and root cause of this issue were > shown below. > *Error* > The stack size specified is too small, Specify at least 328k > Error: Could not create the Java Virtual Machine. > Error: A fatal exception has occurred. Program will exit. > *Version* > Cassandra 2.1.21 2.2.15 3.0.19 3.11.5 > *Environment* > $ lscpu > Architecture: aarch64 > Byte Order: Little Endian > $ uname -m > aarch64 > $ java -version > openjdk version "1.8.0_181" > OpenJDK Runtime Environment (build 1.8.0_181-b13) > OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode) > [root@localhost apache-cassandra-3.11.5]# cat /etc/os-release > $ cat /etc/os-release > NAME="CentOS Linux" > VERSION="7 (AltArch)" > ID="centos" > ID_LIKE="rhel fedora" > VERSION_ID="7" > PRETTY_NAME="CentOS Linux 7 (AltArch)" > ANSI_COLOR="0;31" > CPE_NAME="cpe:/o:centos:centos:7" > HOME_URL="https://www.centos.org/; > BUG_REPORT_URL="https://bugs.centos.org/; > *Root Cause* > Checked openjdk-1.8.0 source code, the min stack size is calculated by > StackYellowPage, StackRedPage, StackShadowPage, OS page size. Among those > parameters, *default OS page size of aarch64 CentOS 7.6 is 64K, however > aarch64 Ubuntu 18.04 and X86 CentOS are both 4K.* > This difference causes JVM on aarch64 Ubuntu 18.04 needs 164K per-thread > stack size, but 328K required on aarch64 CentOS 7.6. > The formula is > os::Linux::min_stack_allowed = MAX2(os::Linux::min_stack_allowed, > (size_t)(StackYellowPages+StackRedPages+StackShadowPages) * > Linux::page_size() + > (2*BytesPerWord COMPILER2_PRESENT(+1)) * Linux::vm_default_page_size()); > *Parameters on aarch64 CentOS7.6* > intx StackRedPages = 1 > intx StackShadowPages = 1 > intx StackYellowPages = 1 > pageSize 64K > BytesPerWord 8 > vm_default_page_size 8K > As a result, we have min_stack_allowed = (1 + 1 + 1) * 64K + (2 * 8 + 1) * 8K > = 328K > > I could see some similar issues asked for specified achitecture, but no root > cause analyzed. I hope this could help you decide proper stack size for all > common OS. > If you have any suggestion, pls let me know. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15358) Cassandra alpha 4 testing - Nodes crashing due to bufferpool allocator issue
[
https://issues.apache.org/jira/browse/CASSANDRA-15358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992601#comment-16992601
]
Benedict Elliott Smith edited comment on CASSANDRA-15358 at 12/10/19 2:32 PM:
--
I'm afraid I don't know the answer to either question. I'm waiting for a
review for the fix to be merged into trunk, and maybe I can try to accelerate
that by poking some colleagues. However a stable release of 4.0 is probably
months away, given current community resources and where they are focused.
nb: it's worth noting this delay is partially attributable to the community
aiming for a much lower defect rate in the first GA release of this major
version, and a large amount of effort is currently being invested in automated
testing. Depending on your prior approach to GA releases you might be
comfortable treating a beta or RC release as usable, which I hope will come
much sooner.
was (Author: benedict):
I'm afraid I don't know the answer to either question. I'm waiting for a
review for the fix to be merged into trunk, and maybe I can try to accelerate
that by poking some colleagues. However a stable release of 4.0 is probably
months away, given current community resources and where they are focused.
> Cassandra alpha 4 testing - Nodes crashing due to bufferpool allocator issue
>
>
> Key: CASSANDRA-15358
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15358
> Project: Cassandra
> Issue Type: Bug
> Components: Test/benchmark
>Reporter: Santhosh Kumar Ramalingam
>Assignee: Benedict Elliott Smith
>Priority: Normal
> Labels: 4.0, alpha
> Attachments: all_errors.txt, verbose_logs.diff, verbose_logs.txt
>
>
> Hitting a bug with cassandra 4 alpha version. The same bug is repeated with
> difefrent version of Java(8,11 &12) [~benedict]
>
> Stack trace:
> {code:java}
> INFO [main] 2019-10-11 16:07:12,024 Server.java:164 - Starting listening for
> CQL clients on /1.3.0.6:9042 (unencrypted)...
> WARN [OptionalTasks:1] 2019-10-11 16:07:13,961 CassandraRoleManager.java:343
> - CassandraRoleManager skipped default role setup: some nodes were not ready
> INFO [OptionalTasks:1] 2019-10-11 16:07:13,961 CassandraRoleManager.java:369
> - Setup task failed with error, rescheduling
> WARN [Messaging-EventLoop-3-2] 2019-10-11 16:07:22,038 NoSpamLogger.java:94 -
> 10.3x.4x.5x:7000->1.3.0.5:7000-LARGE_MESSAGES-[no-channel] dropping message
> of type PING_REQ whose timeout expired before reaching the network
> WARN [OptionalTasks:1] 2019-10-11 16:07:23,963 CassandraRoleManager.java:343
> - CassandraRoleManager skipped default role setup: some nodes were not ready
> INFO [OptionalTasks:1] 2019-10-11 16:07:23,963 CassandraRoleManager.java:369
> - Setup task failed with error, rescheduling
> INFO [Messaging-EventLoop-3-6] 2019-10-11 16:07:32,759 NoSpamLogger.java:91 -
> 10.3x.4x.5x:7000->1.3.0.2:7000-URGENT_MESSAGES-[no-channel] failed to connect
> io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..)
> failed: Connection refused: /1.3.0.2:7000
> Caused by: java.net.ConnectException: finishConnect(..) failed: Connection
> refused
> at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124)
> at io.netty.channel.unix.Socket.finishConnect(Socket.java:243)
> at
> io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:667)
> at
> io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:644)
> at
> io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:524)
> at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:414)
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> WARN [Messaging-EventLoop-3-3] 2019-10-11 16:11:32,639 NoSpamLogger.java:94 -
> 1.3.4.6:7000->1.3.4.5:7000-URGENT_MESSAGES-[no-channel] dropping message of
> type GOSSIP_DIGEST_SYN whose timeout expired before reaching the network
> INFO [Messaging-EventLoop-3-18] 2019-10-11 16:11:33,077 NoSpamLogger.java:91
> - 1.3.4.5:7000->1.3.4.4:7000-URGENT_MESSAGES-[no-channel] failed to connect
>
> ERROR [Messaging-EventLoop-3-11] 2019-10-10 01:34:34,407
> InboundMessageHandler.java:657 -
> 1.3.4.5:7000->1.3.4.8:7000-LARGE_MESSAGES-0b7d09cd unexpected exception
> caught while
[jira] [Commented] (CASSANDRA-15358) Cassandra alpha 4 testing - Nodes crashing due to bufferpool allocator issue
[
https://issues.apache.org/jira/browse/CASSANDRA-15358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992601#comment-16992601
]
Benedict Elliott Smith commented on CASSANDRA-15358:
I'm afraid I don't know the answer to either question. I'm waiting for a
review for the fix to be merged into trunk, and maybe I can try to accelerate
that by poking some colleagues. However a stable release of 4.0 is probably
months away, given current community resources and where they are focused.
> Cassandra alpha 4 testing - Nodes crashing due to bufferpool allocator issue
>
>
> Key: CASSANDRA-15358
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15358
> Project: Cassandra
> Issue Type: Bug
> Components: Test/benchmark
>Reporter: Santhosh Kumar Ramalingam
>Assignee: Benedict Elliott Smith
>Priority: Normal
> Labels: 4.0, alpha
> Attachments: all_errors.txt, verbose_logs.diff, verbose_logs.txt
>
>
> Hitting a bug with cassandra 4 alpha version. The same bug is repeated with
> difefrent version of Java(8,11 &12) [~benedict]
>
> Stack trace:
> {code:java}
> INFO [main] 2019-10-11 16:07:12,024 Server.java:164 - Starting listening for
> CQL clients on /1.3.0.6:9042 (unencrypted)...
> WARN [OptionalTasks:1] 2019-10-11 16:07:13,961 CassandraRoleManager.java:343
> - CassandraRoleManager skipped default role setup: some nodes were not ready
> INFO [OptionalTasks:1] 2019-10-11 16:07:13,961 CassandraRoleManager.java:369
> - Setup task failed with error, rescheduling
> WARN [Messaging-EventLoop-3-2] 2019-10-11 16:07:22,038 NoSpamLogger.java:94 -
> 10.3x.4x.5x:7000->1.3.0.5:7000-LARGE_MESSAGES-[no-channel] dropping message
> of type PING_REQ whose timeout expired before reaching the network
> WARN [OptionalTasks:1] 2019-10-11 16:07:23,963 CassandraRoleManager.java:343
> - CassandraRoleManager skipped default role setup: some nodes were not ready
> INFO [OptionalTasks:1] 2019-10-11 16:07:23,963 CassandraRoleManager.java:369
> - Setup task failed with error, rescheduling
> INFO [Messaging-EventLoop-3-6] 2019-10-11 16:07:32,759 NoSpamLogger.java:91 -
> 10.3x.4x.5x:7000->1.3.0.2:7000-URGENT_MESSAGES-[no-channel] failed to connect
> io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..)
> failed: Connection refused: /1.3.0.2:7000
> Caused by: java.net.ConnectException: finishConnect(..) failed: Connection
> refused
> at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124)
> at io.netty.channel.unix.Socket.finishConnect(Socket.java:243)
> at
> io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:667)
> at
> io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:644)
> at
> io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:524)
> at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:414)
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> WARN [Messaging-EventLoop-3-3] 2019-10-11 16:11:32,639 NoSpamLogger.java:94 -
> 1.3.4.6:7000->1.3.4.5:7000-URGENT_MESSAGES-[no-channel] dropping message of
> type GOSSIP_DIGEST_SYN whose timeout expired before reaching the network
> INFO [Messaging-EventLoop-3-18] 2019-10-11 16:11:33,077 NoSpamLogger.java:91
> - 1.3.4.5:7000->1.3.4.4:7000-URGENT_MESSAGES-[no-channel] failed to connect
>
> ERROR [Messaging-EventLoop-3-11] 2019-10-10 01:34:34,407
> InboundMessageHandler.java:657 -
> 1.3.4.5:7000->1.3.4.8:7000-LARGE_MESSAGES-0b7d09cd unexpected exception
> caught while processing inbound messages; terminating connection
> java.lang.IllegalArgumentException: initialBuffer is not a direct buffer.
> at io.netty.buffer.UnpooledDirectByteBuf.(UnpooledDirectByteBuf.java:87)
> at
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:59)
> at
> org.apache.cassandra.net.BufferPoolAllocator$Wrapped.(BufferPoolAllocator.java:95)
> at
> org.apache.cassandra.net.BufferPoolAllocator.newDirectBuffer(BufferPoolAllocator.java:56)
> at
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
> at
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
> at
> io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
[jira] [Commented] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992599#comment-16992599
]
Benedict Elliott Smith commented on CASSANDRA-14355:
This looks like the known issues with repair, which has been partially
addressed in both CASSANDRA-14096 and CASSANDRA-15202.
The best solution to this problem until then is to repair smaller ranges at a
time, or to upgrade to a version with CASSANDRA-14096 (CASSANDRA-15202 won't be
released for a while, though it could easily be back ported, and might
justifiably be considered for that).
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Normal
> Fix For: 3.11.x
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png, LongGC_Dominator-Tree.png, LongGC_Histogram.png,
> LongGC_Problem-Suspect-1_FastThreadLocalThread.png, LongGC_nodetool_info.txt
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15394) Remove list iterators
[
https://issues.apache.org/jira/browse/CASSANDRA-15394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992489#comment-16992489
]
Benedict Elliott Smith commented on CASSANDRA-15394:
(y)
I think there's some value in introducing {{isize}} as a norm, only because
when we have nested loops the norm will have to shift, but it's also pretty
simple for somebody to switch to {{isize}} and {{jsize}} in this case. It
seems fine to use plain {{size}} when there's only one loop.
> Remove list iterators
> -
>
> Key: CASSANDRA-15394
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15394
> Project: Cassandra
> Issue Type: Sub-task
> Components: Local/Compaction
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 4.0
>
>
> We allocate list iterators in several places in hot paths. This converts them
> to get by index. This provides a ~4% improvement in relvant workloads.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15394) Remove list iterators
[ https://issues.apache.org/jira/browse/CASSANDRA-15394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict Elliott Smith updated CASSANDRA-15394: --- Status: Ready to Commit (was: Changes Suggested) > Remove list iterators > - > > Key: CASSANDRA-15394 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15394 > Project: Cassandra > Issue Type: Sub-task > Components: Local/Compaction >Reporter: Blake Eggleston >Assignee: Blake Eggleston >Priority: Normal > Fix For: 4.0 > > > We allocate list iterators in several places in hot paths. This converts them > to get by index. This provides a ~4% improvement in relvant workloads. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992394#comment-16992394
]
Chris Kistner edited comment on CASSANDRA-14355 at 12/10/19 10:17 AM:
--
We have now experienced an issue that might be related to this, however our
Cassandra did not crash yet - it just had frequent (every ~ 2 minutes)
"ConcurrentMarkSweep GC" events of 16+ seconds!
eg.:
{noformat}
WARN [Service Thread] 2019-12-10 08:03:19,969 GCInspector.java:282 -
ConcurrentMarkSweep GC in 19129ms. CMS Old Gen: 7547650016 -> 7547650048; Par
Eden Space: 671088640 -> 251798544; Par Survivor Space: 83886048 -> 0
WARN [Service Thread] 2019-12-10 08:03:37,565 GCInspector.java:282 -
ConcurrentMarkSweep GC in 16379ms. Par Eden Space: 671088640 -> 254509608; Par
Survivor Space: 83886032 -> 0{noformat}
Sometimes it went back down to 200ms again, and after we did a "nodetool drain"
and then removed the node from the cluster the GC time remained sub 250ms.
Our setup is:
* 5 nodes in dc1, 5 nodes in dc2.
* RF: dc1=5, dc2=5
* CL = Local Quorum
* Host with 32GB of RAM -> Cassandra allocates 8GB to heap
* Java version: java-1.8.0-openjdk-1.8.0.151-5.b12
* Using Cassandra Reaper 4.6.1 where we scheduled a repair with 32
segments/node (364 segments in total)
I have attached some screenshots from our~11GB heap dump, where
io.netty.util.concurrent.FastThreadLocalThread contributed towards 6.4GB of the
heap size:
* Problem Suspect 1: LongGC_Problem-Suspect-1_FastThreadLocalThread.png
* Dominator Tree: LongGC_Dominator-Tree.png
* Histogram: LongGC_Histogram.png
I have also attached the output of "nodetool status": LongGC_nodetool_info.txt
We have not tried out Cassandra 3.11.5, which apparently solved the Repair OOME
issue: CASSANDRA-14096
was (Author: padakwaak):
We have now experienced an issue that might be related to this, however our
Cassandra did not crash yet - it just had frequent (every ~ 2 minutes)
"ConcurrentMarkSweep GC" events of 16+ seconds!
eg.:
{noformat}
Line 78776: WARN [Service Thread] 2019-12-10 08:03:19,969 GCInspector.java:282
- ConcurrentMarkSweep GC in 19129ms. CMS Old Gen: 7547650016 -> 7547650048;
Par Eden Space: 671088640 -> 251798544; Par Survivor Space: 83886048 -> 0
Line 79080: WARN [Service Thread] 2019-12-10 08:03:37,565
GCInspector.java:282 - ConcurrentMarkSweep GC in 16379ms. Par Eden Space:
671088640 -> 254509608; Par Survivor Space: 83886032 -> 0{noformat}
Sometimes it went back down to 200ms again, and after we did a "nodetool drain"
and then removed the node from the cluster the GC time remained sub 250ms.
Our setup is:
* 5 nodes in dc1, 5 nodes in dc2.
* RF: dc1=5, dc2=5
* CL = Local Quorum
* Host with 32GB of RAM -> Cassandra allocates 8GB to heap
* Java version: java-1.8.0-openjdk-1.8.0.151-5.b12
* Using Cassandra Reaper 4.6.1 where we scheduled a repair with 32
segments/node (364 segments in total)
I have attached some screenshots from our~11GB heap dump, where
io.netty.util.concurrent.FastThreadLocalThread contributed towards 6.4GB of the
heap size:
* Problem Suspect 1: LongGC_Problem-Suspect-1_FastThreadLocalThread.png
* Dominator Tree: LongGC_Dominator-Tree.png
* Histogram: LongGC_Histogram.png
I have also attached the output of "nodetool status": LongGC_nodetool_info.txt
We have not tried out Cassandra 3.11.5, which apparently solved the Repair OOME
issue: CASSANDRA-14096
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Normal
> Fix For: 3.11.x
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png, LongGC_Dominator-Tree.png, LongGC_Histogram.png,
> LongGC_Problem-Suspect-1_FastThreadLocalThread.png, LongGC_nodetool_info.txt
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992394#comment-16992394
]
Chris Kistner commented on CASSANDRA-14355:
---
We have now experienced an issue that might be related to this, however our
Cassandra did not crash yet - it just had frequent (every ~ 2 minutes)
"ConcurrentMarkSweep GC" events of 16+ seconds!
eg.:
{noformat}
Line 78776: WARN [Service Thread] 2019-12-10 08:03:19,969 GCInspector.java:282
- ConcurrentMarkSweep GC in 19129ms. CMS Old Gen: 7547650016 -> 7547650048;
Par Eden Space: 671088640 -> 251798544; Par Survivor Space: 83886048 -> 0
Line 79080: WARN [Service Thread] 2019-12-10 08:03:37,565
GCInspector.java:282 - ConcurrentMarkSweep GC in 16379ms. Par Eden Space:
671088640 -> 254509608; Par Survivor Space: 83886032 -> 0{noformat}
Sometimes it went back down to 200ms again, and after we did a "nodetool drain"
and then removed the node from the cluster the GC time remained sub 250ms.
Our setup is:
* 5 nodes in dc1, 5 nodes in dc2.
* RF: dc1=5, dc2=5
* CL = Local Quorum
* Host with 32GB of RAM -> Cassandra allocates 8GB to heap
* Java version: java-1.8.0-openjdk-1.8.0.151-5.b12
* Using Cassandra Reaper 4.6.1 where we scheduled a repair with 32
segments/node (364 segments in total)
I have attached some screenshots from our~11GB heap dump, where
io.netty.util.concurrent.FastThreadLocalThread contributed towards 6.4GB of the
heap size:
* Problem Suspect 1: LongGC_Problem-Suspect-1_FastThreadLocalThread.png
* Dominator Tree: LongGC_Dominator-Tree.png
* Histogram: LongGC_Histogram.png
I have also attached the output of "nodetool status": LongGC_nodetool_info.txt
We have not tried out Cassandra 3.11.5, which apparently solved the Repair OOME
issue: CASSANDRA-14096
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Normal
> Fix For: 3.11.x
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png, LongGC_Dominator-Tree.png, LongGC_Histogram.png,
> LongGC_Problem-Suspect-1_FastThreadLocalThread.png, LongGC_nodetool_info.txt
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Kistner updated CASSANDRA-14355:
--
Attachment: LongGC_nodetool_info.txt
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Normal
> Fix For: 3.11.x
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png, LongGC_Dominator-Tree.png, LongGC_Histogram.png,
> LongGC_Problem-Suspect-1_FastThreadLocalThread.png, LongGC_nodetool_info.txt
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Kistner updated CASSANDRA-14355:
--
Attachment: LongGC_Histogram.png
LongGC_Dominator-Tree.png
LongGC_Problem-Suspect-1_FastThreadLocalThread.png
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Normal
> Fix For: 3.11.x
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png, LongGC_Dominator-Tree.png, LongGC_Histogram.png,
> LongGC_Problem-Suspect-1_FastThreadLocalThread.png
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15429) Support NodeTool for in-jvm dtest
[ https://issues.apache.org/jira/browse/CASSANDRA-15429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992286#comment-16992286 ] Alex Petrov commented on CASSANDRA-15429: - Not an official review, just glanced over the patch. Can we hide nodetool behind a feature flag, same as native transport and other things? I can also see a TODO about GCInspector, can you elaborate a bit? > Support NodeTool for in-jvm dtest > - > > Key: CASSANDRA-15429 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15429 > Project: Cassandra > Issue Type: New Feature > Components: Test/dtest >Reporter: Yifan Cai >Assignee: Yifan Cai >Priority: Normal > Labels: pull-request-available > Fix For: 4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > In-JVM dtest framework does not support nodetool as of now. This > functionality is wanted in some tests, e.g. constructing an end-to-end test > scenario that uses nodetool. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
