[jira] [Updated] (CASSANDRA-15584) 4.0 quality testing: Tooling - External Ecosystem
[ https://issues.apache.org/jira/browse/CASSANDRA-15584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jordan West updated CASSANDRA-15584: Status: Awaiting Feedback (was: Triage Needed) [~samt] would it be a good start to list the tools that are going to be covered by this ticket? > 4.0 quality testing: Tooling - External Ecosystem > - > > Key: CASSANDRA-15584 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15584 > Project: Cassandra > Issue Type: Task >Reporter: Josh McKenzie >Assignee: Sam Tunnicliffe >Priority: Normal > Fix For: 4.0-rc > > > Many users of Apache Cassandra employ open source tooling to automate > Cassandra configuration, runtime management, and repair scheduling. Prior to > release, we need to confirm that popular third-party tools such as Reaper, > Priam, etc. function properly. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15585) 4.0 quality testing: Test Frameworks, Tooling, Infra / Automation
[ https://issues.apache.org/jira/browse/CASSANDRA-15585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jordan West updated CASSANDRA-15585: Status: Awaiting Feedback (was: Triage Needed) In its current state this ticket should probably cover work on Harry & Replay. Does that sound right [~ifesdjeen], [~yifanc], [~marcuse], [~cscotta]? > 4.0 quality testing: Test Frameworks, Tooling, Infra / Automation > - > > Key: CASSANDRA-15585 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15585 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Josh McKenzie >Assignee: Jordan West >Priority: Normal > Fix For: 4.0-rc > > > This area refers to contributions to test frameworks/tooling (e.g., dtests, > QuickTheories, CASSANDRA-14821), and automation enabling those tools to be > applied at scale (e.g., replay testing via Spark-based replay of captured FQL > logs). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15598) cassandra-stress in user profile mode fails on handling set> columns
[ https://issues.apache.org/jira/browse/CASSANDRA-15598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jordan West updated CASSANDRA-15598: Fix Version/s: 4.x > cassandra-stress in user profile mode fails on handling set> columns > - > > Key: CASSANDRA-15598 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15598 > Project: Cassandra > Issue Type: Bug > Components: Tool/stress >Reporter: Dmitry Kropachev >Priority: Normal > Fix For: 4.x > > Attachments: cs_no_mv_basic_profile.zip > > > Reason: > c-s reads metadata from database, and build column generator from it. > But this logic is build in a way so that it does not support generator > recursion, which is necessary to support embedded collections. > Fix: > This fix is simple exclude this type of columns from being processed. > Steps to reproduce: > # docker run -d --name cassandra cassandra:latest > # cassandra-stress user profile=cs_no_mv_basic_profile.yaml ops'(insert=1)' > cl=ONE n=20 -pop seq=1..1000 -port jmx=6868 -mode cql3 native -rate > threads=2 -node 172.17.0.2 > # docker exec -ti cassandra cqlsh -e 'ALTER TABLE mview.users ADD ( > asdasdasd set>);' > # cassandra-stress user profile=cs_no_mv_basic_profile.yaml ops'(insert=1)' > cl=ONE n=20 -pop seq=1..1000 -port jmx=6868 -mode cql3 native -rate > threads=2 -node 172.17.0.2 > Result: > Extra Schema Definitions: null > Generator Configs: > password: Size: Fixed: key=80; > last_name: Size: Uniform: min=1,max=32; > id: Size: Uniform: min=1,max=10; > first_name: Size: Fixed: key=16; > email: Size: Uniform: min=16,max=50; > username: Size: Uniform: min=10,max=30; > Query Definitions: > read1: CQL:select * from mview.users where id = ? LIMIT 10;Fields:samerow; > Token Range Queries: > Insert Settings: > partitions: fixed(1) > batchtype: UNLOGGED > Connected to cluster: , max pending requests per connection 128, max > connections per host 8 > Datacenter: datacenter1; Host: /172.17.0.2:9042; Rack: rack1 > Created schema. Sleeping 1s for propagation. > java.lang.NullPointerException > at > org.apache.cassandra.stress.StressProfile$ColumnInfo.getGenerator(StressProfile.java:760) > at > org.apache.cassandra.stress.StressProfile$ColumnInfo.getGenerator(StressProfile.java:800) > at > org.apache.cassandra.stress.StressProfile$ColumnInfo.getGenerator(StressProfile.java:800) > at > org.apache.cassandra.stress.StressProfile$ColumnInfo.getGenerator(StressProfile.java:755) > at > org.apache.cassandra.stress.StressProfile$GeneratorFactory.get(StressProfile.java:733) > at > org.apache.cassandra.stress.StressProfile$GeneratorFactory.newGenerator(StressProfile.java:726) > at > org.apache.cassandra.stress.StressProfile.newGenerator(StressProfile.java:696) > at > org.apache.cassandra.stress.StressProfile.printSettings(StressProfile.java:126) > at > org.apache.cassandra.stress.settings.StressSettings.lambda$printSettings$1(StressSettings.java:311) > at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684) > at > org.apache.cassandra.stress.settings.StressSettings.printSettings(StressSettings.java:311) > at org.apache.cassandra.stress.Stress.run(Stress.java:95) > at org.apache.cassandra.stress.Stress.main(Stress.java:62) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15598) cassandra-stress in user profile mode fails on handling set> columns
[
https://issues.apache.org/jira/browse/CASSANDRA-15598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West updated CASSANDRA-15598:
Bug Category: Parent values: Correctness(12982)Level 1 values: API /
Semantic Implementation(12988)
Complexity: Low Hanging Fruit
Discovered By: User Report
Severity: Normal
Status: Open (was: Triage Needed)
[~dkropachev] I was able to confirm the issue locally. Thanks for your detailed
report. One thing I had to change was the alter table command (its missing the
full type definition: {{ALTER TABLE mview.users ADD ( asdasdasd
set>>);}}.
Are you interested in working up a patch? At a minimum we should reject the
schema with a more appropriate error than NPE.
> cassandra-stress in user profile mode fails on handling set> columns
> -
>
> Key: CASSANDRA-15598
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15598
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/stress
>Reporter: Dmitry Kropachev
>Priority: Normal
> Attachments: cs_no_mv_basic_profile.zip
>
>
> Reason:
> c-s reads metadata from database, and build column generator from it.
> But this logic is build in a way so that it does not support generator
> recursion, which is necessary to support embedded collections.
> Fix:
> This fix is simple exclude this type of columns from being processed.
> Steps to reproduce:
> # docker run -d --name cassandra cassandra:latest
> # cassandra-stress user profile=cs_no_mv_basic_profile.yaml ops'(insert=1)'
> cl=ONE n=20 -pop seq=1..1000 -port jmx=6868 -mode cql3 native -rate
> threads=2 -node 172.17.0.2
> # docker exec -ti cassandra cqlsh -e 'ALTER TABLE mview.users ADD (
> asdasdasd set>);'
> # cassandra-stress user profile=cs_no_mv_basic_profile.yaml ops'(insert=1)'
> cl=ONE n=20 -pop seq=1..1000 -port jmx=6868 -mode cql3 native -rate
> threads=2 -node 172.17.0.2
> Result:
> Extra Schema Definitions: null
> Generator Configs:
> password: Size: Fixed: key=80;
> last_name: Size: Uniform: min=1,max=32;
> id: Size: Uniform: min=1,max=10;
> first_name: Size: Fixed: key=16;
> email: Size: Uniform: min=16,max=50;
> username: Size: Uniform: min=10,max=30;
> Query Definitions:
> read1: CQL:select * from mview.users where id = ? LIMIT 10;Fields:samerow;
> Token Range Queries:
> Insert Settings:
> partitions: fixed(1)
> batchtype: UNLOGGED
> Connected to cluster: , max pending requests per connection 128, max
> connections per host 8
> Datacenter: datacenter1; Host: /172.17.0.2:9042; Rack: rack1
> Created schema. Sleeping 1s for propagation.
> java.lang.NullPointerException
> at
> org.apache.cassandra.stress.StressProfile$ColumnInfo.getGenerator(StressProfile.java:760)
> at
> org.apache.cassandra.stress.StressProfile$ColumnInfo.getGenerator(StressProfile.java:800)
> at
> org.apache.cassandra.stress.StressProfile$ColumnInfo.getGenerator(StressProfile.java:800)
> at
> org.apache.cassandra.stress.StressProfile$ColumnInfo.getGenerator(StressProfile.java:755)
> at
> org.apache.cassandra.stress.StressProfile$GeneratorFactory.get(StressProfile.java:733)
> at
> org.apache.cassandra.stress.StressProfile$GeneratorFactory.newGenerator(StressProfile.java:726)
> at
> org.apache.cassandra.stress.StressProfile.newGenerator(StressProfile.java:696)
> at
> org.apache.cassandra.stress.StressProfile.printSettings(StressProfile.java:126)
> at
> org.apache.cassandra.stress.settings.StressSettings.lambda$printSettings$1(StressSettings.java:311)
> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
> at
> org.apache.cassandra.stress.settings.StressSettings.printSettings(StressSettings.java:311)
> at org.apache.cassandra.stress.Stress.run(Stress.java:95)
> at org.apache.cassandra.stress.Stress.main(Stress.java:62)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15597) Correct Visibility and Improve Safety of Methods in LatencyMetrics
[ https://issues.apache.org/jira/browse/CASSANDRA-15597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051611#comment-17051611 ] Jordan West commented on CASSANDRA-15597: - [~Jeff] patch LGTM. Do you want to also include the visibility changes to add/removeChildren? Thanks. > Correct Visibility and Improve Safety of Methods in LatencyMetrics > -- > > Key: CASSANDRA-15597 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15597 > Project: Cassandra > Issue Type: Improvement > Components: Observability/Metrics >Reporter: Jordan West >Assignee: Jeff >Priority: Normal > Fix For: 4.0 > > > * add/removeChildren does not need to be public (and exposing addChildren is > unsafe since no lock is used). > * casting in the constructor is safer than casting each time in removeChildren -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15604) Remove obsolete table/MV options from schema tables
[
https://issues.apache.org/jira/browse/CASSANDRA-15604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West updated CASSANDRA-15604:
Resolution: Won't Fix
Status: Resolved (was: Awaiting Feedback)
Ok thanks. Closing since there is a more general proposed solution that should
theoretically address the backwards compatibility concerns as well.
> Remove obsolete table/MV options from schema tables
> ---
>
> Key: CASSANDRA-15604
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15604
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Olivier Michallat
>Priority: Normal
>
> CASSANDRA-13910 removed the {{read_repair_chance}} and
> {{dclocal_read_repair_chance}} options for tables and materialized views.
> CASSANDRA-12868 made {{default_time_to_live}} illegal for materialized views.
> However the corresponding columns still exist in {{system_schema.tables}} and
> {{system_schema.views}}. Maybe it would make sense to remove them in
> Cassandra 4. Historically, obsolete options have been removed in similar
> cases, for example {{compaction_strategy_class}},
> {{compaction_strategy_options}} and {{compression_parameters}} were present
> in 2.2 but removed in 3.0 (arguably there was also a major refactoring of
> schema tables during that transition).
> This is not a huge deal, but it would make things a bit easier for the Java
> driver (our metadata exposes a "client-side describe" feature, and currently
> we assume that any column in the schema table must be added to the CREATE
> TABLE statement).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15604) Remove obsolete table/MV options from schema tables
[
https://issues.apache.org/jira/browse/CASSANDRA-15604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West reassigned CASSANDRA-15604:
---
Assignee: Jordan West
> Remove obsolete table/MV options from schema tables
> ---
>
> Key: CASSANDRA-15604
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15604
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Olivier Michallat
>Assignee: Jordan West
>Priority: Normal
>
> CASSANDRA-13910 removed the {{read_repair_chance}} and
> {{dclocal_read_repair_chance}} options for tables and materialized views.
> CASSANDRA-12868 made {{default_time_to_live}} illegal for materialized views.
> However the corresponding columns still exist in {{system_schema.tables}} and
> {{system_schema.views}}. Maybe it would make sense to remove them in
> Cassandra 4. Historically, obsolete options have been removed in similar
> cases, for example {{compaction_strategy_class}},
> {{compaction_strategy_options}} and {{compression_parameters}} were present
> in 2.2 but removed in 3.0 (arguably there was also a major refactoring of
> schema tables during that transition).
> This is not a huge deal, but it would make things a bit easier for the Java
> driver (our metadata exposes a "client-side describe" feature, and currently
> we assume that any column in the schema table must be added to the CREATE
> TABLE statement).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15604) Remove obsolete table/MV options from schema tables
[
https://issues.apache.org/jira/browse/CASSANDRA-15604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West reassigned CASSANDRA-15604:
---
Assignee: (was: Jordan West)
> Remove obsolete table/MV options from schema tables
> ---
>
> Key: CASSANDRA-15604
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15604
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Olivier Michallat
>Priority: Normal
>
> CASSANDRA-13910 removed the {{read_repair_chance}} and
> {{dclocal_read_repair_chance}} options for tables and materialized views.
> CASSANDRA-12868 made {{default_time_to_live}} illegal for materialized views.
> However the corresponding columns still exist in {{system_schema.tables}} and
> {{system_schema.views}}. Maybe it would make sense to remove them in
> Cassandra 4. Historically, obsolete options have been removed in similar
> cases, for example {{compaction_strategy_class}},
> {{compaction_strategy_options}} and {{compression_parameters}} were present
> in 2.2 but removed in 3.0 (arguably there was also a major refactoring of
> schema tables during that transition).
> This is not a huge deal, but it would make things a bit easier for the Java
> driver (our metadata exposes a "client-side describe" feature, and currently
> we assume that any column in the schema table must be added to the CREATE
> TABLE statement).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15604) Remove obsolete table/MV options from schema tables
[
https://issues.apache.org/jira/browse/CASSANDRA-15604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West updated CASSANDRA-15604:
Status: Awaiting Feedback (was: Triage Needed)
[~omichallat] Could removing cause issues for clients that explicitly state
which columns to select (instead of select *)? My concern is that there could
be some issues on upgrade for some clients (which we may not have taken as much
care to check for during the 2.x -> 3.x changes).
Also would https://issues.apache.org/jira/browse/CASSANDRA-14825 address this?
> Remove obsolete table/MV options from schema tables
> ---
>
> Key: CASSANDRA-15604
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15604
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Olivier Michallat
>Priority: Normal
>
> CASSANDRA-13910 removed the {{read_repair_chance}} and
> {{dclocal_read_repair_chance}} options for tables and materialized views.
> CASSANDRA-12868 made {{default_time_to_live}} illegal for materialized views.
> However the corresponding columns still exist in {{system_schema.tables}} and
> {{system_schema.views}}. Maybe it would make sense to remove them in
> Cassandra 4. Historically, obsolete options have been removed in similar
> cases, for example {{compaction_strategy_class}},
> {{compaction_strategy_options}} and {{compression_parameters}} were present
> in 2.2 but removed in 3.0 (arguably there was also a major refactoring of
> schema tables during that transition).
> This is not a huge deal, but it would make things a bit easier for the Java
> driver (our metadata exposes a "client-side describe" feature, and currently
> we assume that any column in the schema table must be added to the CREATE
> TABLE statement).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15611) Build and Test with both Java 8 & 11 in Circle CI
[ https://issues.apache.org/jira/browse/CASSANDRA-15611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jordan West updated CASSANDRA-15611: Change Category: Quality Assurance Complexity: Low Hanging Fruit Status: Open (was: Triage Needed) > Build and Test with both Java 8 & 11 in Circle CI > - > > Key: CASSANDRA-15611 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15611 > Project: Cassandra > Issue Type: Improvement > Components: Sidecar >Reporter: Jon Haddad >Assignee: Jon Haddad >Priority: Normal > > We currently only build and test with Java 8. We should ensure Java 11 is > fully supported for both builds and testing in CircleCI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15597) Correct Visibility and Improve Safety of Methods in LatencyMetrics
[ https://issues.apache.org/jira/browse/CASSANDRA-15597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17049384#comment-17049384 ] Jordan West commented on CASSANDRA-15597: - Thanks for the patch [~jefftt]. I will review soon. > Correct Visibility and Improve Safety of Methods in LatencyMetrics > -- > > Key: CASSANDRA-15597 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15597 > Project: Cassandra > Issue Type: Improvement > Components: Observability/Metrics >Reporter: Jordan West >Assignee: Jeff >Priority: Normal > Fix For: 4.0 > > > * add/removeChildren does not need to be public (and exposing addChildren is > unsafe since no lock is used). > * casting in the constructor is safer than casting each time in removeChildren -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15597) Correct Visibility and Improve Safety of Methods in LatencyMetrics
[ https://issues.apache.org/jira/browse/CASSANDRA-15597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jordan West updated CASSANDRA-15597: Reviewers: Jordan West, Jordan West (was: Jordan West) Jordan West, Jordan West Status: Review In Progress (was: Patch Available) > Correct Visibility and Improve Safety of Methods in LatencyMetrics > -- > > Key: CASSANDRA-15597 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15597 > Project: Cassandra > Issue Type: Improvement > Components: Observability/Metrics >Reporter: Jordan West >Assignee: Jeff >Priority: Normal > Fix For: 4.0 > > > * add/removeChildren does not need to be public (and exposing addChildren is > unsafe since no lock is used). > * casting in the constructor is safer than casting each time in removeChildren -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15597) Correct Visibility and Improve Safety of Methods in LatencyMetrics
[ https://issues.apache.org/jira/browse/CASSANDRA-15597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jordan West updated CASSANDRA-15597: Test and Documentation Plan: existing test suite Status: Patch Available (was: In Progress) > Correct Visibility and Improve Safety of Methods in LatencyMetrics > -- > > Key: CASSANDRA-15597 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15597 > Project: Cassandra > Issue Type: Improvement > Components: Observability/Metrics >Reporter: Jordan West >Assignee: Jeff >Priority: Normal > Fix For: 4.0 > > > * add/removeChildren does not need to be public (and exposing addChildren is > unsafe since no lock is used). > * casting in the constructor is safer than casting each time in removeChildren -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15597) Correct Visibility and Improve Safety of Methods in LatencyMetrics
[ https://issues.apache.org/jira/browse/CASSANDRA-15597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jordan West reassigned CASSANDRA-15597: --- Assignee: Jordan West > Correct Visibility and Improve Safety of Methods in LatencyMetrics > -- > > Key: CASSANDRA-15597 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15597 > Project: Cassandra > Issue Type: Improvement > Components: Observability/Metrics >Reporter: Jordan West >Assignee: Jordan West >Priority: Normal > Fix For: 4.0 > > > * add/removeChildren does not need to be public (and exposing addChildren is > unsafe since no lock is used). > * casting in the constructor is safer than casting each time in removeChildren -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15597) Correct Visibility and Improve Safety of Methods in LatencyMetrics
[ https://issues.apache.org/jira/browse/CASSANDRA-15597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jordan West reassigned CASSANDRA-15597: --- Assignee: Jeff (was: Jordan West) > Correct Visibility and Improve Safety of Methods in LatencyMetrics > -- > > Key: CASSANDRA-15597 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15597 > Project: Cassandra > Issue Type: Improvement > Components: Observability/Metrics >Reporter: Jordan West >Assignee: Jeff >Priority: Normal > Fix For: 4.0 > > > * add/removeChildren does not need to be public (and exposing addChildren is > unsafe since no lock is used). > * casting in the constructor is safer than casting each time in removeChildren -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15256) Clean up redundant nodetool commands added in 4.0
[
https://issues.apache.org/jira/browse/CASSANDRA-15256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West updated CASSANDRA-15256:
Status: Ready to Commit (was: Review In Progress)
LGTM. +1. Some notes for future readers:
{{getReplicas}}:
- Was added as part of Transient Replication patch (CASSANDRA-14404)
- Eventually calls {{getNaturalReplicasForToken(String, ByteBuffer)}}
function just like {{getNaturalEndpoints}} does but with an additional check
that the table exists (although it uses no metadata from this table)
- Does not include ports like {{getendpoints}} does. Although
{{getendpoints}} uses the deprecated method {{getNaturalEndpoints}} it has an
option to use the replacement {{getNaturalEndpointsWithPort}}
Other Changes:
- HandoffWindow (added in CASSANDRA-13728) was duplicate of GetMaxHintWindow
(added in CASSANDRA-11720) which is more conventional naming
- Removes duplicate DisableAuditLog listing
> Clean up redundant nodetool commands added in 4.0
> -
>
> Key: CASSANDRA-15256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15256
> Project: Cassandra
> Issue Type: Improvement
> Components: Tool/nodetool
>Reporter: Chris Lohfink
>Assignee: Chris Lohfink
>Priority: Low
> Fix For: 4.0
>
>
> Both hintedhandoff and getendpoints had a 2nd command added that does the
> exact thing but rewritten (and in getReplicas not as well) not just aliased
> (like cf->tablestats). Also a minor cleanup is same command added multiple
> times to nodetool command list. We should clean this up before 4.0 release
> before people become reliant on the newly introduced command name. If we want
> them renamed as that we should rename and link with alias like we do with
> cf->table others.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15256) Clean up redundant nodetool commands added in 4.0
[ https://issues.apache.org/jira/browse/CASSANDRA-15256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jordan West updated CASSANDRA-15256: Reviewers: Jordan West, Jordan West (was: Jordan West) Jordan West, Jordan West Status: Review In Progress (was: Patch Available) > Clean up redundant nodetool commands added in 4.0 > - > > Key: CASSANDRA-15256 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15256 > Project: Cassandra > Issue Type: Improvement > Components: Tool/nodetool >Reporter: Chris Lohfink >Assignee: Chris Lohfink >Priority: Low > Fix For: 4.0 > > > Both hintedhandoff and getendpoints had a 2nd command added that does the > exact thing but rewritten (and in getReplicas not as well) not just aliased > (like cf->tablestats). Also a minor cleanup is same command added multiple > times to nodetool command list. We should clean this up before 4.0 release > before people become reliant on the newly introduced command name. If we want > them renamed as that we should rename and link with alias like we do with > cf->table others. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15533) Don't allocate unneeded MergeIterator in OnDiskToken#iterator
[ https://issues.apache.org/jira/browse/CASSANDRA-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jordan West updated CASSANDRA-15533: Fix Version/s: 4.0 > Don't allocate unneeded MergeIterator in OnDiskToken#iterator > -- > > Key: CASSANDRA-15533 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15533 > Project: Cassandra > Issue Type: Improvement > Components: Feature/SASI >Reporter: Jordan West >Assignee: Jordan West >Priority: Normal > Fix For: 4.0, 4.0-rc > > > When reviewing CASSANDRA-15392 it became apparent that the MergeIterator > allocated by OnDiskToken#iterator is rarely necessary and so we should avoid > allocating one when not needed and skip the MergeIterator pool when needed > because its unlikely to be sized correctly. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15560) Change io.compressor.LZ4Compressor to LZ4SafeDecompressor
[ https://issues.apache.org/jira/browse/CASSANDRA-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jordan West updated CASSANDRA-15560: Fix Version/s: 4.0 > Change io.compressor.LZ4Compressor to LZ4SafeDecompressor > - > > Key: CASSANDRA-15560 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15560 > Project: Cassandra > Issue Type: Improvement > Components: Feature/Compression >Reporter: Jordan West >Assignee: Jordan West >Priority: Normal > Fix For: 4.0, 4.0-rc > > > CASSANDRA-15556 and related tickets showed that LZ4FastDecompressor can crash > the JVM and that LZ4SafeDecompressor performs better w/o the crash risk — its > also not deprecated. While we protect ourselves by checksumming the > compressed data but that doesn’t mean we should leave deprecated code that > can segfault the jvm (providing a potential DDOS vector among other things) > in crucial places like io.compress. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15597) Correct Visibility and Improve Safety of Methods in LatencyMetrics
[ https://issues.apache.org/jira/browse/CASSANDRA-15597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jordan West updated CASSANDRA-15597: Change Category: Code Clarity Complexity: Low Hanging Fruit Fix Version/s: 4.0 Status: Open (was: Triage Needed) > Correct Visibility and Improve Safety of Methods in LatencyMetrics > -- > > Key: CASSANDRA-15597 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15597 > Project: Cassandra > Issue Type: Improvement > Components: Observability/Metrics >Reporter: Jordan West >Priority: Normal > Fix For: 4.0 > > > * add/removeChildren does not need to be public (and exposing addChildren is > unsafe since no lock is used). > * casting in the constructor is safer than casting each time in removeChildren -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15597) Correct Visibility and Improve Safety of Methods in LatencyMetrics
Jordan West created CASSANDRA-15597: --- Summary: Correct Visibility and Improve Safety of Methods in LatencyMetrics Key: CASSANDRA-15597 URL: https://issues.apache.org/jira/browse/CASSANDRA-15597 Project: Cassandra Issue Type: Improvement Components: Observability/Metrics Reporter: Jordan West * add/removeChildren does not need to be public (and exposing addChildren is unsafe since no lock is used). * casting in the constructor is safer than casting each time in removeChildren -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15596) Make Gossiper.aVeryLongTime configurable
Jordan West created CASSANDRA-15596: --- Summary: Make Gossiper.aVeryLongTime configurable Key: CASSANDRA-15596 URL: https://issues.apache.org/jira/browse/CASSANDRA-15596 Project: Cassandra Issue Type: Improvement Components: Cluster/Gossip Reporter: Jordan West This is useful for testing primarily and operator control maybe -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15213) DecayingEstimatedHistogramReservoir Inefficiencies
[
https://issues.apache.org/jira/browse/CASSANDRA-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West updated CASSANDRA-15213:
Status: Ready to Commit (was: Review In Progress)
> DecayingEstimatedHistogramReservoir Inefficiencies
> --
>
> Key: CASSANDRA-15213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15213
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Benedict Elliott Smith
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: 15213-perf-branch, 15213-perf-trunk
>
>
> * {{LongAdder}} introduced to trunk consumes 9MiB of heap without user
> schemas, and this will grow significantly under contention and user schemas
> with many tables. This is because {{LongAdder}} is a very heavy class
> designed for single contended values.
> ** This can likely be improved significantly, without significant loss of
> performance in the contended case, by simply increasing the size of our
> primitive backing array and providing multiple buckets, with each thread
> picking a bucket to increment, or simply multiple backing arrays. Probably a
> better way still to do this would be to introduce some competition detection
> to the update, much like {{LongAdder}} utilises, that increases the number of
> backing arrays under competition.
> ** To save memory this approach could partition the space into chunks that
> are likely to be updated together, so that we do not need to duplicate the
> entire array under competition.
> * Similarly, binary search is costly and a measurable cost as a share of the
> new networking work (without filtering it was > 10% of the CPU used overall).
> We can compute an approximation floor(log2 n / log2 1.2) extremely cheaply,
> to save the random memory access costs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15213) DecayingEstimatedHistogramReservoir Inefficiencies
[
https://issues.apache.org/jira/browse/CASSANDRA-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West updated CASSANDRA-15213:
Test and Documentation Plan: Existing microbenchmarks, new cluster
benchmarks, quick theories tests
Status: Patch Available (was: Open)
> DecayingEstimatedHistogramReservoir Inefficiencies
> --
>
> Key: CASSANDRA-15213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15213
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Benedict Elliott Smith
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: 15213-perf-branch, 15213-perf-trunk
>
>
> * {{LongAdder}} introduced to trunk consumes 9MiB of heap without user
> schemas, and this will grow significantly under contention and user schemas
> with many tables. This is because {{LongAdder}} is a very heavy class
> designed for single contended values.
> ** This can likely be improved significantly, without significant loss of
> performance in the contended case, by simply increasing the size of our
> primitive backing array and providing multiple buckets, with each thread
> picking a bucket to increment, or simply multiple backing arrays. Probably a
> better way still to do this would be to introduce some competition detection
> to the update, much like {{LongAdder}} utilises, that increases the number of
> backing arrays under competition.
> ** To save memory this approach could partition the space into chunks that
> are likely to be updated together, so that we do not need to duplicate the
> entire array under competition.
> * Similarly, binary search is costly and a measurable cost as a share of the
> new networking work (without filtering it was > 10% of the CPU used overall).
> We can compute an approximation floor(log2 n / log2 1.2) extremely cheaply,
> to save the random memory access costs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15213) DecayingEstimatedHistogramReservoir Inefficiencies
[
https://issues.apache.org/jira/browse/CASSANDRA-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West updated CASSANDRA-15213:
Reviewers: Benedict Elliott Smith, Jordan West (was: Chris Lohfink)
Status: Review In Progress (was: Patch Available)
> DecayingEstimatedHistogramReservoir Inefficiencies
> --
>
> Key: CASSANDRA-15213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15213
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Benedict Elliott Smith
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: 15213-perf-branch, 15213-perf-trunk
>
>
> * {{LongAdder}} introduced to trunk consumes 9MiB of heap without user
> schemas, and this will grow significantly under contention and user schemas
> with many tables. This is because {{LongAdder}} is a very heavy class
> designed for single contended values.
> ** This can likely be improved significantly, without significant loss of
> performance in the contended case, by simply increasing the size of our
> primitive backing array and providing multiple buckets, with each thread
> picking a bucket to increment, or simply multiple backing arrays. Probably a
> better way still to do this would be to introduce some competition detection
> to the update, much like {{LongAdder}} utilises, that increases the number of
> backing arrays under competition.
> ** To save memory this approach could partition the space into chunks that
> are likely to be updated together, so that we do not need to duplicate the
> entire array under competition.
> * Similarly, binary search is costly and a measurable cost as a share of the
> new networking work (without filtering it was > 10% of the CPU used overall).
> We can compute an approximation floor(log2 n / log2 1.2) extremely cheaply,
> to save the random memory access costs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15213) DecayingEstimatedHistogramReservoir Inefficiencies
[
https://issues.apache.org/jira/browse/CASSANDRA-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039297#comment-17039297
]
Jordan West commented on CASSANDRA-15213:
-
Updated the default and the commit message.
[branch|https://github.com/jrwest/cassandra/commits/jwest/15213]
> DecayingEstimatedHistogramReservoir Inefficiencies
> --
>
> Key: CASSANDRA-15213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15213
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Benedict Elliott Smith
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: 15213-perf-branch, 15213-perf-trunk
>
>
> * {{LongAdder}} introduced to trunk consumes 9MiB of heap without user
> schemas, and this will grow significantly under contention and user schemas
> with many tables. This is because {{LongAdder}} is a very heavy class
> designed for single contended values.
> ** This can likely be improved significantly, without significant loss of
> performance in the contended case, by simply increasing the size of our
> primitive backing array and providing multiple buckets, with each thread
> picking a bucket to increment, or simply multiple backing arrays. Probably a
> better way still to do this would be to introduce some competition detection
> to the update, much like {{LongAdder}} utilises, that increases the number of
> backing arrays under competition.
> ** To save memory this approach could partition the space into chunks that
> are likely to be updated together, so that we do not need to duplicate the
> entire array under competition.
> * Similarly, binary search is costly and a measurable cost as a share of the
> new networking work (without filtering it was > 10% of the CPU used overall).
> We can compute an approximation floor(log2 n / log2 1.2) extremely cheaply,
> to save the random memory access costs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15556) When a LZ4 stream is corrupted it could cause the JVM to crash
[
https://issues.apache.org/jira/browse/CASSANDRA-15556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034571#comment-17034571
]
Jordan West commented on CASSANDRA-15556:
-
LGTM. Thanks [~dcapwell]
> When a LZ4 stream is corrupted it could cause the JVM to crash
> --
>
> Key: CASSANDRA-15556
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15556
> Project: Cassandra
> Issue Type: Sub-task
> Components: Messaging/Client
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-alpha
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This is a spin off of CASSANDRA-15313 and CASSANDRA-15299
> This was found when lz4 sees compressed data (not all, but happens) which is
> corrupted; in some cases the JVM crashes producing the following
> {code}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGBUS (0xa) at pc=0x000110d46ad1, pid=86555, tid=0x1103
> #
> # JRE version: OpenJDK Runtime Environment (8.0_222-b10) (build 1.8.0_222-b10)
> # Java VM: OpenJDK 64-Bit Server VM (25.222-b10 mixed mode bsd-amd64
> compressed oops)
> # Problematic frame:
> # C [liblz4-java2766366422904460658.dylib+0x3ad1] LZ4_decompress_fast+0xf1
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /Users/davidcapwell/src/github/apache/cassandra/hs_err_pid86555.log
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15526) Fix flakey test - org.apache.cassandra.index.sasi.SASIIndexTest testConcurrentMemtableReadsAndWrites
[
https://issues.apache.org/jira/browse/CASSANDRA-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034531#comment-17034531
]
Jordan West commented on CASSANDRA-15526:
-
[~e.dimitrova] the use of {{size()}} in the code is definitely suspicious given
the javadoc. Whats odd to me is {{isEmpty}}, {{first}}, {{size}} all rely on an
internal method called {{findFirst}}. Unless we are deleting from the set,
which right now I don't believe we are, it seems unexpected that the first two
would return indicating elements are present but {{size}} does not -- even with
the javadoc warning. With that in mind I'd like to better understand the root
cause of the failure. I also explored if we could get rid of using {{size}} but
I don't see a clear path to doing so at this time.
> Fix flakey test - org.apache.cassandra.index.sasi.SASIIndexTest
> testConcurrentMemtableReadsAndWrites
>
>
> Key: CASSANDRA-15526
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15526
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: David Capwell
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> {code}
> junit.framework.AssertionFailedError
> at
> org.apache.cassandra.index.sasi.utils.RangeIterator.(RangeIterator.java:46)
> at
> org.apache.cassandra.index.sasi.memory.KeyRangeIterator.(KeyRangeIterator.java:42)
> at
> org.apache.cassandra.index.sasi.memory.TrieMemIndex$ConcurrentTrie.search(TrieMemIndex.java:150)
> at
> org.apache.cassandra.index.sasi.memory.TrieMemIndex.search(TrieMemIndex.java:102)
> at
> org.apache.cassandra.index.sasi.memory.IndexMemtable.search(IndexMemtable.java:70)
> at
> org.apache.cassandra.index.sasi.conf.ColumnIndex.searchMemtable(ColumnIndex.java:138)
> at org.apache.cassandra.index.sasi.TermIterator.build(TermIterator.java:91)
> at
> org.apache.cassandra.index.sasi.plan.QueryController.getIndexes(QueryController.java:145)
> at
> org.apache.cassandra.index.sasi.plan.Operation$Builder.complete(Operation.java:434)
> at org.apache.cassandra.index.sasi.plan.QueryPlan.analyze(QueryPlan.java:57)
> at org.apache.cassandra.index.sasi.plan.QueryPlan.execute(QueryPlan.java:68)
> at
> org.apache.cassandra.index.sasi.SASIIndex.lambda$searcherFor$2(SASIIndex.java:301)
> at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:455)
> at
> org.apache.cassandra.index.sasi.SASIIndexTest.getIndexed(SASIIndexTest.java:2576)
> at
> org.apache.cassandra.index.sasi.SASIIndexTest.getPaged(SASIIndexTest.java:2537)
> at
> org.apache.cassandra.index.sasi.SASIIndexTest.testConcurrentMemtableReadsAndWrites(SASIIndexTest.java:1108)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15560) Change io.compressor.LZ4Compressor to LZ4SafeDecompressor
[ https://issues.apache.org/jira/browse/CASSANDRA-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jordan West updated CASSANDRA-15560: Change Category: Quality Assurance Complexity: Low Hanging Fruit Fix Version/s: 4.0-rc Status: Open (was: Triage Needed) > Change io.compressor.LZ4Compressor to LZ4SafeDecompressor > - > > Key: CASSANDRA-15560 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15560 > Project: Cassandra > Issue Type: Improvement > Components: Feature/Compression >Reporter: Jordan West >Assignee: Jordan West >Priority: Normal > Fix For: 4.0-rc > > > CASSANDRA-15556 and related tickets showed that LZ4FastDecompressor can crash > the JVM and that LZ4SafeDecompressor performs better w/o the crash risk — its > also not deprecated. While we protect ourselves by checksumming the > compressed data but that doesn’t mean we should leave deprecated code that > can segfault the jvm (providing a potential DDOS vector among other things) > in crucial places like io.compress. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15560) Change io.compressor.LZ4Compressor to LZ4SafeDecompressor
Jordan West created CASSANDRA-15560: --- Summary: Change io.compressor.LZ4Compressor to LZ4SafeDecompressor Key: CASSANDRA-15560 URL: https://issues.apache.org/jira/browse/CASSANDRA-15560 Project: Cassandra Issue Type: Improvement Components: Feature/Compression Reporter: Jordan West Assignee: Jordan West CASSANDRA-15556 and related tickets showed that LZ4FastDecompressor can crash the JVM and that LZ4SafeDecompressor performs better w/o the crash risk — its also not deprecated. While we protect ourselves by checksumming the compressed data but that doesn’t mean we should leave deprecated code that can segfault the jvm (providing a potential DDOS vector among other things) in crucial places like io.compress. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15519) Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032658#comment-17032658 ] Jordan West commented on CASSANDRA-15519: - [~clohfink] thanks for committing. Committed in da90439f0052c3a05aaf6d4268a8c719e10fafde > Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at > runtime > - > > Key: CASSANDRA-15519 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15519 > Project: Cassandra > Issue Type: Improvement > Components: Local/Config, Messaging/Client >Reporter: Jordan West >Assignee: Jordan West >Priority: Normal > Labels: pull-request-available > Fix For: 4.0-beta > > Time Spent: 10m > Remaining Estimate: 0h > > CASSANDRA-15013 added configurable global and per endpoint limits on the > number of in flight requests, measured in bytes. During times when the > cluster is overloaded, it can be useful to change this setting without having > to restart the Cassandra process. Changing the limits should affect all > existing and new connections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15526) Fix flakey test - org.apache.cassandra.index.sasi.SASIIndexTest testConcurrentMemtableReadsAndWrites
[
https://issues.apache.org/jira/browse/CASSANDRA-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West reassigned CASSANDRA-15526:
---
Assignee: (was: Jordan West)
> Fix flakey test - org.apache.cassandra.index.sasi.SASIIndexTest
> testConcurrentMemtableReadsAndWrites
>
>
> Key: CASSANDRA-15526
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15526
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: David Capwell
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> {code}
> junit.framework.AssertionFailedError
> at
> org.apache.cassandra.index.sasi.utils.RangeIterator.(RangeIterator.java:46)
> at
> org.apache.cassandra.index.sasi.memory.KeyRangeIterator.(KeyRangeIterator.java:42)
> at
> org.apache.cassandra.index.sasi.memory.TrieMemIndex$ConcurrentTrie.search(TrieMemIndex.java:150)
> at
> org.apache.cassandra.index.sasi.memory.TrieMemIndex.search(TrieMemIndex.java:102)
> at
> org.apache.cassandra.index.sasi.memory.IndexMemtable.search(IndexMemtable.java:70)
> at
> org.apache.cassandra.index.sasi.conf.ColumnIndex.searchMemtable(ColumnIndex.java:138)
> at org.apache.cassandra.index.sasi.TermIterator.build(TermIterator.java:91)
> at
> org.apache.cassandra.index.sasi.plan.QueryController.getIndexes(QueryController.java:145)
> at
> org.apache.cassandra.index.sasi.plan.Operation$Builder.complete(Operation.java:434)
> at org.apache.cassandra.index.sasi.plan.QueryPlan.analyze(QueryPlan.java:57)
> at org.apache.cassandra.index.sasi.plan.QueryPlan.execute(QueryPlan.java:68)
> at
> org.apache.cassandra.index.sasi.SASIIndex.lambda$searcherFor$2(SASIIndex.java:301)
> at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:455)
> at
> org.apache.cassandra.index.sasi.SASIIndexTest.getIndexed(SASIIndexTest.java:2576)
> at
> org.apache.cassandra.index.sasi.SASIIndexTest.getPaged(SASIIndexTest.java:2537)
> at
> org.apache.cassandra.index.sasi.SASIIndexTest.testConcurrentMemtableReadsAndWrites(SASIIndexTest.java:1108)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15526) Fix flakey test - org.apache.cassandra.index.sasi.SASIIndexTest testConcurrentMemtableReadsAndWrites
[
https://issues.apache.org/jira/browse/CASSANDRA-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West reassigned CASSANDRA-15526:
---
Assignee: Jordan West
> Fix flakey test - org.apache.cassandra.index.sasi.SASIIndexTest
> testConcurrentMemtableReadsAndWrites
>
>
> Key: CASSANDRA-15526
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15526
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: David Capwell
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> {code}
> junit.framework.AssertionFailedError
> at
> org.apache.cassandra.index.sasi.utils.RangeIterator.(RangeIterator.java:46)
> at
> org.apache.cassandra.index.sasi.memory.KeyRangeIterator.(KeyRangeIterator.java:42)
> at
> org.apache.cassandra.index.sasi.memory.TrieMemIndex$ConcurrentTrie.search(TrieMemIndex.java:150)
> at
> org.apache.cassandra.index.sasi.memory.TrieMemIndex.search(TrieMemIndex.java:102)
> at
> org.apache.cassandra.index.sasi.memory.IndexMemtable.search(IndexMemtable.java:70)
> at
> org.apache.cassandra.index.sasi.conf.ColumnIndex.searchMemtable(ColumnIndex.java:138)
> at org.apache.cassandra.index.sasi.TermIterator.build(TermIterator.java:91)
> at
> org.apache.cassandra.index.sasi.plan.QueryController.getIndexes(QueryController.java:145)
> at
> org.apache.cassandra.index.sasi.plan.Operation$Builder.complete(Operation.java:434)
> at org.apache.cassandra.index.sasi.plan.QueryPlan.analyze(QueryPlan.java:57)
> at org.apache.cassandra.index.sasi.plan.QueryPlan.execute(QueryPlan.java:68)
> at
> org.apache.cassandra.index.sasi.SASIIndex.lambda$searcherFor$2(SASIIndex.java:301)
> at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:455)
> at
> org.apache.cassandra.index.sasi.SASIIndexTest.getIndexed(SASIIndexTest.java:2576)
> at
> org.apache.cassandra.index.sasi.SASIIndexTest.getPaged(SASIIndexTest.java:2537)
> at
> org.apache.cassandra.index.sasi.SASIIndexTest.testConcurrentMemtableReadsAndWrites(SASIIndexTest.java:1108)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15526) Fix flakey test - org.apache.cassandra.index.sasi.SASIIndexTest testConcurrentMemtableReadsAndWrites
[
https://issues.apache.org/jira/browse/CASSANDRA-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032647#comment-17032647
]
Jordan West commented on CASSANDRA-15526:
-
[~e.dimitrova] forgot to mention that you may find this minor change helpful:
https://github.com/jrwest/cassandra/tree/sasi-concurrent-read-write-failure.
Its the added assertion message above.
> Fix flakey test - org.apache.cassandra.index.sasi.SASIIndexTest
> testConcurrentMemtableReadsAndWrites
>
>
> Key: CASSANDRA-15526
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15526
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: David Capwell
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> {code}
> junit.framework.AssertionFailedError
> at
> org.apache.cassandra.index.sasi.utils.RangeIterator.(RangeIterator.java:46)
> at
> org.apache.cassandra.index.sasi.memory.KeyRangeIterator.(KeyRangeIterator.java:42)
> at
> org.apache.cassandra.index.sasi.memory.TrieMemIndex$ConcurrentTrie.search(TrieMemIndex.java:150)
> at
> org.apache.cassandra.index.sasi.memory.TrieMemIndex.search(TrieMemIndex.java:102)
> at
> org.apache.cassandra.index.sasi.memory.IndexMemtable.search(IndexMemtable.java:70)
> at
> org.apache.cassandra.index.sasi.conf.ColumnIndex.searchMemtable(ColumnIndex.java:138)
> at org.apache.cassandra.index.sasi.TermIterator.build(TermIterator.java:91)
> at
> org.apache.cassandra.index.sasi.plan.QueryController.getIndexes(QueryController.java:145)
> at
> org.apache.cassandra.index.sasi.plan.Operation$Builder.complete(Operation.java:434)
> at org.apache.cassandra.index.sasi.plan.QueryPlan.analyze(QueryPlan.java:57)
> at org.apache.cassandra.index.sasi.plan.QueryPlan.execute(QueryPlan.java:68)
> at
> org.apache.cassandra.index.sasi.SASIIndex.lambda$searcherFor$2(SASIIndex.java:301)
> at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:455)
> at
> org.apache.cassandra.index.sasi.SASIIndexTest.getIndexed(SASIIndexTest.java:2576)
> at
> org.apache.cassandra.index.sasi.SASIIndexTest.getPaged(SASIIndexTest.java:2537)
> at
> org.apache.cassandra.index.sasi.SASIIndexTest.testConcurrentMemtableReadsAndWrites(SASIIndexTest.java:1108)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15313) Fix flaky - ChecksummingTransformerTest - org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
[
https://issues.apache.org/jira/browse/CASSANDRA-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032582#comment-17032582
]
Jordan West commented on CASSANDRA-15313:
-
agreed re: keeping test running [~dcapwell]
> Fix flaky - ChecksummingTransformerTest -
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
> ---
>
> Key: CASSANDRA-15313
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15313
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Vinay Chella
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 4.0-alpha
>
> Attachments: CASSANDRA-15313-hack.patch
>
>
> During the recent runs, this test appears to be flaky.
> Example failure:
> [https://circleci.com/gh/vinaykumarchella/cassandra/459#tests/containers/94]
> corruptionCausesFailure-compression -
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
> {code:java}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
> at org.quicktheories.impl.Precursor.(Precursor.java:17)
> at
> org.quicktheories.impl.ConcreteDetachedSource.(ConcreteDetachedSource.java:8)
> at
> org.quicktheories.impl.ConcreteDetachedSource.detach(ConcreteDetachedSource.java:23)
> at org.quicktheories.generators.Retry.generate(CodePoints.java:51)
> at
> org.quicktheories.generators.Generate.lambda$intArrays$10(Generate.java:190)
> at
> org.quicktheories.generators.Generate$$Lambda$17/1847008471.generate(Unknown
> Source)
> at org.quicktheories.core.DescribingGenerator.generate(Gen.java:255)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$mix$10(Gen.java:184)
> at org.quicktheories.core.Gen$$Lambda$45/802243390.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$flatMap$5(Gen.java:93)
> at org.quicktheories.core.Gen$$Lambda$48/363509958.generate(Unknown
> Source)
> at
> org.quicktheories.dsl.TheoryBuilder4.lambda$prgnToTuple$12(TheoryBuilder4.java:188)
> at
> org.quicktheories.dsl.TheoryBuilder4$$Lambda$40/2003496028.generate(Unknown
> Source)
> at org.quicktheories.core.DescribingGenerator.generate(Gen.java:255)
> at org.quicktheories.core.FilteredGenerator.generate(Gen.java:225)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.impl.Core.generate(Core.java:150)
> at org.quicktheories.impl.Core.shrink(Core.java:103)
> at org.quicktheories.impl.Core.run(Core.java:39)
> at org.quicktheories.impl.TheoryRunner.check(TheoryRunner.java:35)
> at org.quicktheories.dsl.TheoryBuilder4.check(TheoryBuilder4.java:150)
> at
> org.quicktheories.dsl.TheoryBuilder4.checkAssert(TheoryBuilder4.java:162)
> at
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest.corruptionCausesFailure(ChecksummingTransformerTest.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15526) Fix flakey test - org.apache.cassandra.index.sasi.SASIIndexTest testConcurrentMemtableReadsAndWrites
[
https://issues.apache.org/jira/browse/CASSANDRA-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032579#comment-17032579
]
Jordan West commented on CASSANDRA-15526:
-
[~e.dimitrova] any input here is appreciated. I hadn't assigned it to myself
since I wasn't able to look at it immediately (likely early next week I can)
but I have been thinking about it on and off. The challenge, at first glance,
is that {{ConcurrentSkipListSet#size()}} doesn't seem to be reliable (and the
javadoc hints that this is sort of the case -- although I can't explain what I
am seeing in that assert above just yet). The problem is we rely on the count
(which is what we use {{size()}} for) for one important thing in
RangeIntersectionIterator, the heuristic to determine if we use bounce or
lookup.
> Fix flakey test - org.apache.cassandra.index.sasi.SASIIndexTest
> testConcurrentMemtableReadsAndWrites
>
>
> Key: CASSANDRA-15526
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15526
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: David Capwell
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> {code}
> junit.framework.AssertionFailedError
> at
> org.apache.cassandra.index.sasi.utils.RangeIterator.(RangeIterator.java:46)
> at
> org.apache.cassandra.index.sasi.memory.KeyRangeIterator.(KeyRangeIterator.java:42)
> at
> org.apache.cassandra.index.sasi.memory.TrieMemIndex$ConcurrentTrie.search(TrieMemIndex.java:150)
> at
> org.apache.cassandra.index.sasi.memory.TrieMemIndex.search(TrieMemIndex.java:102)
> at
> org.apache.cassandra.index.sasi.memory.IndexMemtable.search(IndexMemtable.java:70)
> at
> org.apache.cassandra.index.sasi.conf.ColumnIndex.searchMemtable(ColumnIndex.java:138)
> at org.apache.cassandra.index.sasi.TermIterator.build(TermIterator.java:91)
> at
> org.apache.cassandra.index.sasi.plan.QueryController.getIndexes(QueryController.java:145)
> at
> org.apache.cassandra.index.sasi.plan.Operation$Builder.complete(Operation.java:434)
> at org.apache.cassandra.index.sasi.plan.QueryPlan.analyze(QueryPlan.java:57)
> at org.apache.cassandra.index.sasi.plan.QueryPlan.execute(QueryPlan.java:68)
> at
> org.apache.cassandra.index.sasi.SASIIndex.lambda$searcherFor$2(SASIIndex.java:301)
> at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:455)
> at
> org.apache.cassandra.index.sasi.SASIIndexTest.getIndexed(SASIIndexTest.java:2576)
> at
> org.apache.cassandra.index.sasi.SASIIndexTest.getPaged(SASIIndexTest.java:2537)
> at
> org.apache.cassandra.index.sasi.SASIIndexTest.testConcurrentMemtableReadsAndWrites(SASIIndexTest.java:1108)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15556) When a LZ4 stream is corrupted it could cause the JVM to crash
[
https://issues.apache.org/jira/browse/CASSANDRA-15556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032249#comment-17032249
]
Jordan West edited comment on CASSANDRA-15556 at 2/7/20 9:52 AM:
-
Thanks [~dcapwell] -- appreciate the benchmarks and new unit test! Patch looks
good overall. A couple comments:
* I think we need to update the dependency in {{lib}} as well.
* Minor nit: any reason I'm missing why the test can't be included in the
existing class (it already has some unit tests)?
Also re:
bq. it has been fixed already for sstables it seems.
I agree they are separate code paths but it looks like we are still using
{{FastDecompressor}} in
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/compress/LZ4Compressor.java#L75
EDIT: I now see the patch you linked in cassandra-dev slack. But should we be
using a deprecated lz4 function at all?
was (Author: jrwest):
Thanks [~dcapwell] -- appreciate the benchmarks and new unit test! Patch looks
good overall. A couple comments:
* I think we need to update the dependency in {{lib}} as well.
* Minor nit: any reason I'm missing why the test can't be included in the
existing class (it already has some unit tests)?
Also re:
bq. it has been fixed already for sstables it seems.
I agree they are separate code paths but it looks like we are still using
{{FastDecompressor}} in
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/compress/LZ4Compressor.java#L75
> When a LZ4 stream is corrupted it could cause the JVM to crash
> --
>
> Key: CASSANDRA-15556
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15556
> Project: Cassandra
> Issue Type: Sub-task
> Components: Messaging/Client
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-alpha
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This is a spin off of CASSANDRA-15313 and CASSANDRA-15299
> This was found when lz4 sees compressed data (not all, but happens) which is
> corrupted; in some cases the JVM crashes producing the following
> {code}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGBUS (0xa) at pc=0x000110d46ad1, pid=86555, tid=0x1103
> #
> # JRE version: OpenJDK Runtime Environment (8.0_222-b10) (build 1.8.0_222-b10)
> # Java VM: OpenJDK 64-Bit Server VM (25.222-b10 mixed mode bsd-amd64
> compressed oops)
> # Problematic frame:
> # C [liblz4-java2766366422904460658.dylib+0x3ad1] LZ4_decompress_fast+0xf1
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /Users/davidcapwell/src/github/apache/cassandra/hs_err_pid86555.log
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15313) Fix flaky - ChecksummingTransformerTest - org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
[
https://issues.apache.org/jira/browse/CASSANDRA-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032254#comment-17032254
]
Jordan West commented on CASSANDRA-15313:
-
If we aren't able to fix it now / are waiting for a future change, can we mark
it as flaky (maybe this JIRA being open is fine) in some way but not @Ignore
it? I think that would more likely ensure we don't forget it -- especially
since its been effective at finding bugs. The test should apply to the code
post-15529 as well.
> Fix flaky - ChecksummingTransformerTest -
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
> ---
>
> Key: CASSANDRA-15313
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15313
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Vinay Chella
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 4.0-alpha
>
> Attachments: CASSANDRA-15313-hack.patch
>
>
> During the recent runs, this test appears to be flaky.
> Example failure:
> [https://circleci.com/gh/vinaykumarchella/cassandra/459#tests/containers/94]
> corruptionCausesFailure-compression -
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
> {code:java}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
> at org.quicktheories.impl.Precursor.(Precursor.java:17)
> at
> org.quicktheories.impl.ConcreteDetachedSource.(ConcreteDetachedSource.java:8)
> at
> org.quicktheories.impl.ConcreteDetachedSource.detach(ConcreteDetachedSource.java:23)
> at org.quicktheories.generators.Retry.generate(CodePoints.java:51)
> at
> org.quicktheories.generators.Generate.lambda$intArrays$10(Generate.java:190)
> at
> org.quicktheories.generators.Generate$$Lambda$17/1847008471.generate(Unknown
> Source)
> at org.quicktheories.core.DescribingGenerator.generate(Gen.java:255)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$mix$10(Gen.java:184)
> at org.quicktheories.core.Gen$$Lambda$45/802243390.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$flatMap$5(Gen.java:93)
> at org.quicktheories.core.Gen$$Lambda$48/363509958.generate(Unknown
> Source)
> at
> org.quicktheories.dsl.TheoryBuilder4.lambda$prgnToTuple$12(TheoryBuilder4.java:188)
> at
> org.quicktheories.dsl.TheoryBuilder4$$Lambda$40/2003496028.generate(Unknown
> Source)
> at org.quicktheories.core.DescribingGenerator.generate(Gen.java:255)
> at org.quicktheories.core.FilteredGenerator.generate(Gen.java:225)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.impl.Core.generate(Core.java:150)
> at org.quicktheories.impl.Core.shrink(Core.java:103)
> at org.quicktheories.impl.Core.run(Core.java:39)
> at org.quicktheories.impl.TheoryRunner.check(TheoryRunner.java:35)
> at org.quicktheories.dsl.TheoryBuilder4.check(TheoryBuilder4.java:150)
> at
> org.quicktheories.dsl.TheoryBuilder4.checkAssert(TheoryBuilder4.java:162)
> at
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest.corruptionCausesFailure(ChecksummingTransformerTest.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15556) When a LZ4 stream is corrupted it could cause the JVM to crash
[
https://issues.apache.org/jira/browse/CASSANDRA-15556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032249#comment-17032249
]
Jordan West edited comment on CASSANDRA-15556 at 2/7/20 9:39 AM:
-
Thanks [~dcapwell] -- appreciate the benchmarks and new unit test! Patch looks
good overall. A couple comments:
* I think we need to update the dependency in {{lib}} as well.
* Minor nit: any reason I'm missing why the test can't be included in the
existing class (it already has some unit tests)?
Also re:
bq. it has been fixed already for sstables it seems.
I agree they are separate code paths but it looks like we are still using
{{FastDecompressor}} in
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/compress/LZ4Compressor.java#L75
was (Author: jrwest):
Thanks [~dcapwell]. Patch overall looks good a couple comments:
* I think we need to update the dependency in {{lib}} as well.
* Minor nit: any reason I'm missing why the test can't be included in the
existing class (it already has some unit tests)?
Also re:
bq. it has been fixed already for sstables it seems.
I agree they are separate code paths but it looks like we are still using
{{FastDecompressor}} in
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/compress/LZ4Compressor.java#L75
> When a LZ4 stream is corrupted it could cause the JVM to crash
> --
>
> Key: CASSANDRA-15556
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15556
> Project: Cassandra
> Issue Type: Sub-task
> Components: Messaging/Client
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-alpha
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This is a spin off of CASSANDRA-15313 and CASSANDRA-15299
> This was found when lz4 sees compressed data (not all, but happens) which is
> corrupted; in some cases the JVM crashes producing the following
> {code}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGBUS (0xa) at pc=0x000110d46ad1, pid=86555, tid=0x1103
> #
> # JRE version: OpenJDK Runtime Environment (8.0_222-b10) (build 1.8.0_222-b10)
> # Java VM: OpenJDK 64-Bit Server VM (25.222-b10 mixed mode bsd-amd64
> compressed oops)
> # Problematic frame:
> # C [liblz4-java2766366422904460658.dylib+0x3ad1] LZ4_decompress_fast+0xf1
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /Users/davidcapwell/src/github/apache/cassandra/hs_err_pid86555.log
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15556) When a LZ4 stream is corrupted it could cause the JVM to crash
[
https://issues.apache.org/jira/browse/CASSANDRA-15556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West updated CASSANDRA-15556:
Reviewers: Jordan West, Jordan West (was: Jordan West)
Jordan West, Jordan West
Status: Review In Progress (was: Patch Available)
> When a LZ4 stream is corrupted it could cause the JVM to crash
> --
>
> Key: CASSANDRA-15556
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15556
> Project: Cassandra
> Issue Type: Sub-task
> Components: Messaging/Client
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-alpha
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This is a spin off of CASSANDRA-15313 and CASSANDRA-15299
> This was found when lz4 sees compressed data (not all, but happens) which is
> corrupted; in some cases the JVM crashes producing the following
> {code}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGBUS (0xa) at pc=0x000110d46ad1, pid=86555, tid=0x1103
> #
> # JRE version: OpenJDK Runtime Environment (8.0_222-b10) (build 1.8.0_222-b10)
> # Java VM: OpenJDK 64-Bit Server VM (25.222-b10 mixed mode bsd-amd64
> compressed oops)
> # Problematic frame:
> # C [liblz4-java2766366422904460658.dylib+0x3ad1] LZ4_decompress_fast+0xf1
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /Users/davidcapwell/src/github/apache/cassandra/hs_err_pid86555.log
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15556) When a LZ4 stream is corrupted it could cause the JVM to crash
[
https://issues.apache.org/jira/browse/CASSANDRA-15556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032249#comment-17032249
]
Jordan West commented on CASSANDRA-15556:
-
Thanks [~dcapwell]. Patch overall looks good a couple comments:
* I think we need to update the dependency in {{lib}} as well.
* Minor nit: any reason I'm missing why the test can't be included in the
existing class (it already has some unit tests)?
Also re:
bq. it has been fixed already for sstables it seems.
I agree they are separate code paths but it looks like we are still using
{{FastDecompressor}} in
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/compress/LZ4Compressor.java#L75
> When a LZ4 stream is corrupted it could cause the JVM to crash
> --
>
> Key: CASSANDRA-15556
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15556
> Project: Cassandra
> Issue Type: Sub-task
> Components: Messaging/Client
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-alpha
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This is a spin off of CASSANDRA-15313 and CASSANDRA-15299
> This was found when lz4 sees compressed data (not all, but happens) which is
> corrupted; in some cases the JVM crashes producing the following
> {code}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGBUS (0xa) at pc=0x000110d46ad1, pid=86555, tid=0x1103
> #
> # JRE version: OpenJDK Runtime Environment (8.0_222-b10) (build 1.8.0_222-b10)
> # Java VM: OpenJDK 64-Bit Server VM (25.222-b10 mixed mode bsd-amd64
> compressed oops)
> # Problematic frame:
> # C [liblz4-java2766366422904460658.dylib+0x3ad1] LZ4_decompress_fast+0xf1
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /Users/davidcapwell/src/github/apache/cassandra/hs_err_pid86555.log
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15555) Lower the amount of garbage ChecksummingTransformerTest generates by reusing memory
[ https://issues.apache.org/jira/browse/CASSANDRA-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jordan West updated CASSANDRA-1: Reviewers: Brandon Williams, Jordan West (was: Brandon Williams, David Capwell) > Lower the amount of garbage ChecksummingTransformerTest generates by reusing > memory > --- > > Key: CASSANDRA-1 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1 > Project: Cassandra > Issue Type: Sub-task > Components: Test/unit >Reporter: David Capwell >Assignee: David Capwell >Priority: Normal > Labels: pull-request-available > Fix For: 4.0-alpha > > Time Spent: 10m > Remaining Estimate: 0h > > When the test finds a issue and attempts to shrink, a ton of garbage is > generated which cause the test to then fail with GC issues. Since the test > is a quick theories test, we can rewrite the generates to reuse memory rather > than creating new objects for each iteration. This will also let us move > away from String which requires multiple copies in the test and gets rid of > the getBytes() call which is dependent on the system its called on. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15555) Lower the amount of garbage ChecksummingTransformerTest generates by reusing memory
[ https://issues.apache.org/jira/browse/CASSANDRA-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032060#comment-17032060 ] Jordan West commented on CASSANDRA-1: - LGTM. +1. Thanks David. > Lower the amount of garbage ChecksummingTransformerTest generates by reusing > memory > --- > > Key: CASSANDRA-1 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1 > Project: Cassandra > Issue Type: Sub-task > Components: Test/unit >Reporter: David Capwell >Assignee: David Capwell >Priority: Normal > Labels: pull-request-available > Fix For: 4.0-alpha > > Time Spent: 10m > Remaining Estimate: 0h > > When the test finds a issue and attempts to shrink, a ton of garbage is > generated which cause the test to then fail with GC issues. Since the test > is a quick theories test, we can rewrite the generates to reuse memory rather > than creating new objects for each iteration. This will also let us move > away from String which requires multiple copies in the test and gets rid of > the getBytes() call which is dependent on the system its called on. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Issue Comment Deleted] (CASSANDRA-15313) Fix flaky - ChecksummingTransformerTest - org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
[
https://issues.apache.org/jira/browse/CASSANDRA-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West updated CASSANDRA-15313:
Comment: was deleted
(was: LGTM. +1. Thanks David! )
> Fix flaky - ChecksummingTransformerTest -
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
> ---
>
> Key: CASSANDRA-15313
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15313
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Vinay Chella
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 4.0-alpha
>
> Attachments: CASSANDRA-15313-hack.patch
>
>
> During the recent runs, this test appears to be flaky.
> Example failure:
> [https://circleci.com/gh/vinaykumarchella/cassandra/459#tests/containers/94]
> corruptionCausesFailure-compression -
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
> {code:java}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
> at org.quicktheories.impl.Precursor.(Precursor.java:17)
> at
> org.quicktheories.impl.ConcreteDetachedSource.(ConcreteDetachedSource.java:8)
> at
> org.quicktheories.impl.ConcreteDetachedSource.detach(ConcreteDetachedSource.java:23)
> at org.quicktheories.generators.Retry.generate(CodePoints.java:51)
> at
> org.quicktheories.generators.Generate.lambda$intArrays$10(Generate.java:190)
> at
> org.quicktheories.generators.Generate$$Lambda$17/1847008471.generate(Unknown
> Source)
> at org.quicktheories.core.DescribingGenerator.generate(Gen.java:255)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$mix$10(Gen.java:184)
> at org.quicktheories.core.Gen$$Lambda$45/802243390.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$flatMap$5(Gen.java:93)
> at org.quicktheories.core.Gen$$Lambda$48/363509958.generate(Unknown
> Source)
> at
> org.quicktheories.dsl.TheoryBuilder4.lambda$prgnToTuple$12(TheoryBuilder4.java:188)
> at
> org.quicktheories.dsl.TheoryBuilder4$$Lambda$40/2003496028.generate(Unknown
> Source)
> at org.quicktheories.core.DescribingGenerator.generate(Gen.java:255)
> at org.quicktheories.core.FilteredGenerator.generate(Gen.java:225)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.impl.Core.generate(Core.java:150)
> at org.quicktheories.impl.Core.shrink(Core.java:103)
> at org.quicktheories.impl.Core.run(Core.java:39)
> at org.quicktheories.impl.TheoryRunner.check(TheoryRunner.java:35)
> at org.quicktheories.dsl.TheoryBuilder4.check(TheoryBuilder4.java:150)
> at
> org.quicktheories.dsl.TheoryBuilder4.checkAssert(TheoryBuilder4.java:162)
> at
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest.corruptionCausesFailure(ChecksummingTransformerTest.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15313) Fix flaky - ChecksummingTransformerTest - org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
[
https://issues.apache.org/jira/browse/CASSANDRA-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032059#comment-17032059
]
Jordan West commented on CASSANDRA-15313:
-
LGTM. +1. Thanks David!
> Fix flaky - ChecksummingTransformerTest -
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
> ---
>
> Key: CASSANDRA-15313
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15313
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Vinay Chella
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 4.0-alpha
>
> Attachments: CASSANDRA-15313-hack.patch
>
>
> During the recent runs, this test appears to be flaky.
> Example failure:
> [https://circleci.com/gh/vinaykumarchella/cassandra/459#tests/containers/94]
> corruptionCausesFailure-compression -
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
> {code:java}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
> at org.quicktheories.impl.Precursor.(Precursor.java:17)
> at
> org.quicktheories.impl.ConcreteDetachedSource.(ConcreteDetachedSource.java:8)
> at
> org.quicktheories.impl.ConcreteDetachedSource.detach(ConcreteDetachedSource.java:23)
> at org.quicktheories.generators.Retry.generate(CodePoints.java:51)
> at
> org.quicktheories.generators.Generate.lambda$intArrays$10(Generate.java:190)
> at
> org.quicktheories.generators.Generate$$Lambda$17/1847008471.generate(Unknown
> Source)
> at org.quicktheories.core.DescribingGenerator.generate(Gen.java:255)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$mix$10(Gen.java:184)
> at org.quicktheories.core.Gen$$Lambda$45/802243390.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$flatMap$5(Gen.java:93)
> at org.quicktheories.core.Gen$$Lambda$48/363509958.generate(Unknown
> Source)
> at
> org.quicktheories.dsl.TheoryBuilder4.lambda$prgnToTuple$12(TheoryBuilder4.java:188)
> at
> org.quicktheories.dsl.TheoryBuilder4$$Lambda$40/2003496028.generate(Unknown
> Source)
> at org.quicktheories.core.DescribingGenerator.generate(Gen.java:255)
> at org.quicktheories.core.FilteredGenerator.generate(Gen.java:225)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.impl.Core.generate(Core.java:150)
> at org.quicktheories.impl.Core.shrink(Core.java:103)
> at org.quicktheories.impl.Core.run(Core.java:39)
> at org.quicktheories.impl.TheoryRunner.check(TheoryRunner.java:35)
> at org.quicktheories.dsl.TheoryBuilder4.check(TheoryBuilder4.java:150)
> at
> org.quicktheories.dsl.TheoryBuilder4.checkAssert(TheoryBuilder4.java:162)
> at
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest.corruptionCausesFailure(ChecksummingTransformerTest.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15554) Set QT_SHRINKS=0 project wide
[ https://issues.apache.org/jira/browse/CASSANDRA-15554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031995#comment-17031995 ] Jordan West commented on CASSANDRA-15554: - +1. Thanks! > Set QT_SHRINKS=0 project wide > - > > Key: CASSANDRA-15554 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15554 > Project: Cassandra > Issue Type: Improvement > Components: Test/unit >Reporter: Jordan West >Assignee: Brandon Williams >Priority: Normal > > For many of our tests QuickTheories shrinker is not sufficient and costs > time and memory when shrinking large problem spaces. Disable it by default. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15554) Set QT_SHRINKS=0 project wide
[ https://issues.apache.org/jira/browse/CASSANDRA-15554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jordan West updated CASSANDRA-15554: Reviewers: David Capwell, Jordan West (was: David Capwell) > Set QT_SHRINKS=0 project wide > - > > Key: CASSANDRA-15554 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15554 > Project: Cassandra > Issue Type: Improvement > Components: Test/unit >Reporter: Jordan West >Assignee: Brandon Williams >Priority: Normal > > For many of our tests QuickTheories shrinker is not sufficient and costs > time and memory when shrinking large problem spaces. Disable it by default. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15554) Set QT_SHRINKS=0 project wide
[ https://issues.apache.org/jira/browse/CASSANDRA-15554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031983#comment-17031983 ] Jordan West commented on CASSANDRA-15554: - Noticed there was a minor whitespace issue when I pulled it down. Should we also update [https://github.com/apache/cassandra/blob/trunk/ide/idea/workspace.xml#L170] so its picked up in IntelliJ? > Set QT_SHRINKS=0 project wide > - > > Key: CASSANDRA-15554 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15554 > Project: Cassandra > Issue Type: Improvement > Components: Test/unit >Reporter: Jordan West >Assignee: Brandon Williams >Priority: Normal > > For many of our tests QuickTheories shrinker is not sufficient and costs > time and memory when shrinking large problem spaces. Disable it by default. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15213) DecayingEstimatedHistogramReservoir Inefficiencies
[
https://issues.apache.org/jira/browse/CASSANDRA-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031972#comment-17031972
]
Jordan West edited comment on CASSANDRA-15213 at 2/6/20 9:57 PM:
-
Was asked to run tlp-stress against this branch. After several runs I think at
the cluster level its pretty much a wash between this branch and trunk (except
the memory savings of this branch ofc). Added the output from one of the runs
if folks want to look closer. Its worth noting: this was pretty small / limited
hardware.
Workload:
{code}
./bin/tlp-stress run KeyValue --populate 500k --partitions 1m --threads 4 \
--replication "{'class': 'NetworkTopologyStrategy', 'DC1': 3 }" \
--duration 1h --username $CUSER --password $CPASS \
--host $HOST
{code}
was (Author: jrwest):
Was asked to run tlp-stress against this branch. After several runs I think at
the cluster level its pretty much a wash between this branch and trunk (except
the memory savings of this branch ofc). Added the output from one of the runs
if folks want to look closer. Its worth noting: this was pretty small / limited
hardware.
> DecayingEstimatedHistogramReservoir Inefficiencies
> --
>
> Key: CASSANDRA-15213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15213
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Benedict Elliott Smith
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: 15213-perf-branch, 15213-perf-trunk
>
>
> * {{LongAdder}} introduced to trunk consumes 9MiB of heap without user
> schemas, and this will grow significantly under contention and user schemas
> with many tables. This is because {{LongAdder}} is a very heavy class
> designed for single contended values.
> ** This can likely be improved significantly, without significant loss of
> performance in the contended case, by simply increasing the size of our
> primitive backing array and providing multiple buckets, with each thread
> picking a bucket to increment, or simply multiple backing arrays. Probably a
> better way still to do this would be to introduce some competition detection
> to the update, much like {{LongAdder}} utilises, that increases the number of
> backing arrays under competition.
> ** To save memory this approach could partition the space into chunks that
> are likely to be updated together, so that we do not need to duplicate the
> entire array under competition.
> * Similarly, binary search is costly and a measurable cost as a share of the
> new networking work (without filtering it was > 10% of the CPU used overall).
> We can compute an approximation floor(log2 n / log2 1.2) extremely cheaply,
> to save the random memory access costs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15213) DecayingEstimatedHistogramReservoir Inefficiencies
[
https://issues.apache.org/jira/browse/CASSANDRA-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031972#comment-17031972
]
Jordan West commented on CASSANDRA-15213:
-
Was asked to run tlp-stress against this branch. After several runs I think at
the cluster level its pretty much a wash between this branch and trunk (except
the memory savings of this branch ofc). Added the output from one of the runs
if folks want to look closer. Its worth noting: this was pretty small / limited
hardware.
> DecayingEstimatedHistogramReservoir Inefficiencies
> --
>
> Key: CASSANDRA-15213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15213
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Benedict Elliott Smith
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: 15213-perf-branch, 15213-perf-trunk
>
>
> * {{LongAdder}} introduced to trunk consumes 9MiB of heap without user
> schemas, and this will grow significantly under contention and user schemas
> with many tables. This is because {{LongAdder}} is a very heavy class
> designed for single contended values.
> ** This can likely be improved significantly, without significant loss of
> performance in the contended case, by simply increasing the size of our
> primitive backing array and providing multiple buckets, with each thread
> picking a bucket to increment, or simply multiple backing arrays. Probably a
> better way still to do this would be to introduce some competition detection
> to the update, much like {{LongAdder}} utilises, that increases the number of
> backing arrays under competition.
> ** To save memory this approach could partition the space into chunks that
> are likely to be updated together, so that we do not need to duplicate the
> entire array under competition.
> * Similarly, binary search is costly and a measurable cost as a share of the
> new networking work (without filtering it was > 10% of the CPU used overall).
> We can compute an approximation floor(log2 n / log2 1.2) extremely cheaply,
> to save the random memory access costs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15213) DecayingEstimatedHistogramReservoir Inefficiencies
[
https://issues.apache.org/jira/browse/CASSANDRA-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West updated CASSANDRA-15213:
Attachment: 15213-perf-trunk
15213-perf-branch
> DecayingEstimatedHistogramReservoir Inefficiencies
> --
>
> Key: CASSANDRA-15213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15213
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Benedict Elliott Smith
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: 15213-perf-branch, 15213-perf-trunk
>
>
> * {{LongAdder}} introduced to trunk consumes 9MiB of heap without user
> schemas, and this will grow significantly under contention and user schemas
> with many tables. This is because {{LongAdder}} is a very heavy class
> designed for single contended values.
> ** This can likely be improved significantly, without significant loss of
> performance in the contended case, by simply increasing the size of our
> primitive backing array and providing multiple buckets, with each thread
> picking a bucket to increment, or simply multiple backing arrays. Probably a
> better way still to do this would be to introduce some competition detection
> to the update, much like {{LongAdder}} utilises, that increases the number of
> backing arrays under competition.
> ** To save memory this approach could partition the space into chunks that
> are likely to be updated together, so that we do not need to duplicate the
> entire array under competition.
> * Similarly, binary search is costly and a measurable cost as a share of the
> new networking work (without filtering it was > 10% of the CPU used overall).
> We can compute an approximation floor(log2 n / log2 1.2) extremely cheaply,
> to save the random memory access costs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15554) Set QT_SHRINKS=0 project wide
[ https://issues.apache.org/jira/browse/CASSANDRA-15554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jordan West updated CASSANDRA-15554: Change Category: Performance Complexity: Low Hanging Fruit Component/s: Test/unit Status: Open (was: Triage Needed) > Set QT_SHRINKS=0 project wide > - > > Key: CASSANDRA-15554 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15554 > Project: Cassandra > Issue Type: Improvement > Components: Test/unit >Reporter: Jordan West >Priority: Normal > > For many of our tests QuickTheories shrinker is not sufficient and costs > time and memory when shrinking large problem spaces. Disable it by default. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15554) Set QT_SHRINKS=0 project wide
Jordan West created CASSANDRA-15554: --- Summary: Set QT_SHRINKS=0 project wide Key: CASSANDRA-15554 URL: https://issues.apache.org/jira/browse/CASSANDRA-15554 Project: Cassandra Issue Type: Improvement Reporter: Jordan West For many of our tests QuickTheories shrinker is not sufficient and costs time and memory when shrinking large problem spaces. Disable it by default. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15313) Fix flaky - ChecksummingTransformerTest - org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
[
https://issues.apache.org/jira/browse/CASSANDRA-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031723#comment-17031723
]
Jordan West commented on CASSANDRA-15313:
-
Ah sorry misunderstood. Definitely good to keep that around.
> Fix flaky - ChecksummingTransformerTest -
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
> ---
>
> Key: CASSANDRA-15313
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15313
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Vinay Chella
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 4.0-alpha
>
> Attachments: CASSANDRA-15313-hack.patch
>
>
> During the recent runs, this test appears to be flaky.
> Example failure:
> [https://circleci.com/gh/vinaykumarchella/cassandra/459#tests/containers/94]
> corruptionCausesFailure-compression -
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
> {code:java}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
> at org.quicktheories.impl.Precursor.(Precursor.java:17)
> at
> org.quicktheories.impl.ConcreteDetachedSource.(ConcreteDetachedSource.java:8)
> at
> org.quicktheories.impl.ConcreteDetachedSource.detach(ConcreteDetachedSource.java:23)
> at org.quicktheories.generators.Retry.generate(CodePoints.java:51)
> at
> org.quicktheories.generators.Generate.lambda$intArrays$10(Generate.java:190)
> at
> org.quicktheories.generators.Generate$$Lambda$17/1847008471.generate(Unknown
> Source)
> at org.quicktheories.core.DescribingGenerator.generate(Gen.java:255)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$mix$10(Gen.java:184)
> at org.quicktheories.core.Gen$$Lambda$45/802243390.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$flatMap$5(Gen.java:93)
> at org.quicktheories.core.Gen$$Lambda$48/363509958.generate(Unknown
> Source)
> at
> org.quicktheories.dsl.TheoryBuilder4.lambda$prgnToTuple$12(TheoryBuilder4.java:188)
> at
> org.quicktheories.dsl.TheoryBuilder4$$Lambda$40/2003496028.generate(Unknown
> Source)
> at org.quicktheories.core.DescribingGenerator.generate(Gen.java:255)
> at org.quicktheories.core.FilteredGenerator.generate(Gen.java:225)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.impl.Core.generate(Core.java:150)
> at org.quicktheories.impl.Core.shrink(Core.java:103)
> at org.quicktheories.impl.Core.run(Core.java:39)
> at org.quicktheories.impl.TheoryRunner.check(TheoryRunner.java:35)
> at org.quicktheories.dsl.TheoryBuilder4.check(TheoryBuilder4.java:150)
> at
> org.quicktheories.dsl.TheoryBuilder4.checkAssert(TheoryBuilder4.java:162)
> at
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest.corruptionCausesFailure(ChecksummingTransformerTest.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15313) Fix flaky - ChecksummingTransformerTest - org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
[
https://issues.apache.org/jira/browse/CASSANDRA-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031719#comment-17031719
]
Jordan West commented on CASSANDRA-15313:
-
Not a huge fan of fixing the seed. The intention is for the test to search
different permutations each time its run, not a fixed set (the seed is meant to
help debug once you find that failure -- assuming its deterministic). One seed
may find the original bug this test found but not other/future bugs.
Ideally we'd keep the example count high but balancing that against memory
usage and runtime goals/constraints so we don't make the suites unnecessarily
long for code that hasn't changed.
> Fix flaky - ChecksummingTransformerTest -
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
> ---
>
> Key: CASSANDRA-15313
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15313
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Vinay Chella
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 4.0-alpha
>
> Attachments: CASSANDRA-15313-hack.patch
>
>
> During the recent runs, this test appears to be flaky.
> Example failure:
> [https://circleci.com/gh/vinaykumarchella/cassandra/459#tests/containers/94]
> corruptionCausesFailure-compression -
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
> {code:java}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
> at org.quicktheories.impl.Precursor.(Precursor.java:17)
> at
> org.quicktheories.impl.ConcreteDetachedSource.(ConcreteDetachedSource.java:8)
> at
> org.quicktheories.impl.ConcreteDetachedSource.detach(ConcreteDetachedSource.java:23)
> at org.quicktheories.generators.Retry.generate(CodePoints.java:51)
> at
> org.quicktheories.generators.Generate.lambda$intArrays$10(Generate.java:190)
> at
> org.quicktheories.generators.Generate$$Lambda$17/1847008471.generate(Unknown
> Source)
> at org.quicktheories.core.DescribingGenerator.generate(Gen.java:255)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$mix$10(Gen.java:184)
> at org.quicktheories.core.Gen$$Lambda$45/802243390.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$flatMap$5(Gen.java:93)
> at org.quicktheories.core.Gen$$Lambda$48/363509958.generate(Unknown
> Source)
> at
> org.quicktheories.dsl.TheoryBuilder4.lambda$prgnToTuple$12(TheoryBuilder4.java:188)
> at
> org.quicktheories.dsl.TheoryBuilder4$$Lambda$40/2003496028.generate(Unknown
> Source)
> at org.quicktheories.core.DescribingGenerator.generate(Gen.java:255)
> at org.quicktheories.core.FilteredGenerator.generate(Gen.java:225)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.impl.Core.generate(Core.java:150)
> at org.quicktheories.impl.Core.shrink(Core.java:103)
> at org.quicktheories.impl.Core.run(Core.java:39)
> at org.quicktheories.impl.TheoryRunner.check(TheoryRunner.java:35)
> at org.quicktheories.dsl.TheoryBuilder4.check(TheoryBuilder4.java:150)
> at
> org.quicktheories.dsl.TheoryBuilder4.checkAssert(TheoryBuilder4.java:162)
> at
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest.corruptionCausesFailure(ChecksummingTransformerTest.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15544) Fix flakey test - org.apache.cassandra.index.sasi.SASIIndexTest.testMultiExpressionQueriesWhereRowSplitBetweenSSTables
[
https://issues.apache.org/jira/browse/CASSANDRA-15544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031701#comment-17031701
]
Jordan West commented on CASSANDRA-15544:
-
[~brandon.williams] can you say a bit about the env this was reproduced on
(circle/laptop/java8 or 11)? I've managed to reproduce some of the
SASIIndexTest failures locally on java11 but haven't seen this one yet.
> Fix flakey test -
> org.apache.cassandra.index.sasi.SASIIndexTest.testMultiExpressionQueriesWhereRowSplitBetweenSSTables
> --
>
> Key: CASSANDRA-15544
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15544
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Brandon Williams
>Priority: Normal
>
> {code}
> [junit-timeout] Testcase:
> testMultiExpressionQueriesWhereRowSplitBetweenSSTables(org.apache.cassandra.index.sasi.SASIIndexTest):
> FAILED
> [junit-timeout] []
> [junit-timeout] junit.framework.AssertionFailedError: []
> [junit-timeout] at
> org.apache.cassandra.index.sasi.SASIIndexTest.testMultiExpressionQueriesWhereRowSplitBetweenSSTables(SASIIndexTest.java:589)
> [junit-timeout] at
> org.apache.cassandra.index.sasi.SASIIndexTest.testMultiExpressionQueriesWhereRowSplitBetweenSSTables(SASIIndexTest.java:468)
> [junit-timeout]
> [junit-timeout]
> [junit-timeout] Test org.apache.cassandra.index.sasi.SASIIndexTest FAILED
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15544) Fix flakey test - org.apache.cassandra.index.sasi.SASIIndexTest.testMultiExpressionQueriesWhereRowSplitBetweenSSTables
[
https://issues.apache.org/jira/browse/CASSANDRA-15544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West updated CASSANDRA-15544:
Status: Awaiting Feedback (was: Triage Needed)
> Fix flakey test -
> org.apache.cassandra.index.sasi.SASIIndexTest.testMultiExpressionQueriesWhereRowSplitBetweenSSTables
> --
>
> Key: CASSANDRA-15544
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15544
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Brandon Williams
>Priority: Normal
>
> {code}
> [junit-timeout] Testcase:
> testMultiExpressionQueriesWhereRowSplitBetweenSSTables(org.apache.cassandra.index.sasi.SASIIndexTest):
> FAILED
> [junit-timeout] []
> [junit-timeout] junit.framework.AssertionFailedError: []
> [junit-timeout] at
> org.apache.cassandra.index.sasi.SASIIndexTest.testMultiExpressionQueriesWhereRowSplitBetweenSSTables(SASIIndexTest.java:589)
> [junit-timeout] at
> org.apache.cassandra.index.sasi.SASIIndexTest.testMultiExpressionQueriesWhereRowSplitBetweenSSTables(SASIIndexTest.java:468)
> [junit-timeout]
> [junit-timeout]
> [junit-timeout] Test org.apache.cassandra.index.sasi.SASIIndexTest FAILED
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15527) Fix flakey test - org.apache.cassandra.index.sasi.SASIIndexTest testCrossSSTableQueries
[
https://issues.apache.org/jira/browse/CASSANDRA-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031694#comment-17031694
]
Jordan West commented on CASSANDRA-15527:
-
[~e.dimitrova] it took me 40-50 runs to reproduce -- running the command
[~dcapwell] shared above.
> Fix flakey test - org.apache.cassandra.index.sasi.SASIIndexTest
> testCrossSSTableQueries
> ---
>
> Key: CASSANDRA-15527
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15527
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: David Capwell
>Assignee: Ekaterina Dimitrova
>Priority: Normal
> Fix For: 4.0-alpha
>
> Attachments: CASSANDRA-15527.txt
>
>
> {code}
> junit.framework.AssertionFailedError: [key13, key2977, key2978, key2979,
> key2980, key2982, key2983, key2984, key2985, key6]
> at
> org.apache.cassandra.index.sasi.SASIIndexTest.testCrossSSTableQueries(SASIIndexTest.java:340)
> at
> org.apache.cassandra.index.sasi.SASIIndexTest.testCrossSSTableQueries(SASIIndexTest.java:286)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15313) Fix flaky - ChecksummingTransformerTest - org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
[
https://issues.apache.org/jira/browse/CASSANDRA-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031680#comment-17031680
]
Jordan West edited comment on CASSANDRA-15313 at 2/6/20 3:49 PM:
-
[~samt] actually included the test on my behalf. Its not surprising that this
is a memory issue. Thats one motivation for
https://issues.apache.org/jira/browse/CASSANDRA-15348 which we could convert to
when ready. But in the meantime let me see what we can do to reduce the memory
usage of the test.
Interesting its also potentially finding an issue with LZ4.
was (Author: jrwest):
[~samt] actually included the test on my behalf. Its not surprising that this
is a memory issue. Thats one motivation for
https://issues.apache.org/jira/browse/CASSANDRA-15348 which we could convert to
when ready. But in the meantime let me see what we can do to reduce the memory
usage of the test.
Interesting its also potentially finding an issue with LZ4. Have we filed a
separate JIRA for that?
> Fix flaky - ChecksummingTransformerTest -
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
> ---
>
> Key: CASSANDRA-15313
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15313
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Vinay Chella
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 4.0-alpha
>
> Attachments: CASSANDRA-15313-hack.patch
>
>
> During the recent runs, this test appears to be flaky.
> Example failure:
> [https://circleci.com/gh/vinaykumarchella/cassandra/459#tests/containers/94]
> corruptionCausesFailure-compression -
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
> {code:java}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
> at org.quicktheories.impl.Precursor.(Precursor.java:17)
> at
> org.quicktheories.impl.ConcreteDetachedSource.(ConcreteDetachedSource.java:8)
> at
> org.quicktheories.impl.ConcreteDetachedSource.detach(ConcreteDetachedSource.java:23)
> at org.quicktheories.generators.Retry.generate(CodePoints.java:51)
> at
> org.quicktheories.generators.Generate.lambda$intArrays$10(Generate.java:190)
> at
> org.quicktheories.generators.Generate$$Lambda$17/1847008471.generate(Unknown
> Source)
> at org.quicktheories.core.DescribingGenerator.generate(Gen.java:255)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$mix$10(Gen.java:184)
> at org.quicktheories.core.Gen$$Lambda$45/802243390.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$flatMap$5(Gen.java:93)
> at org.quicktheories.core.Gen$$Lambda$48/363509958.generate(Unknown
> Source)
> at
> org.quicktheories.dsl.TheoryBuilder4.lambda$prgnToTuple$12(TheoryBuilder4.java:188)
> at
> org.quicktheories.dsl.TheoryBuilder4$$Lambda$40/2003496028.generate(Unknown
> Source)
> at org.quicktheories.core.DescribingGenerator.generate(Gen.java:255)
> at org.quicktheories.core.FilteredGenerator.generate(Gen.java:225)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.impl.Core.generate(Core.java:150)
> at org.quicktheories.impl.Core.shrink(Core.java:103)
> at org.quicktheories.impl.Core.run(Core.java:39)
> at org.quicktheories.impl.TheoryRunner.check(TheoryRunner.java:35)
> at org.quicktheories.dsl.TheoryBuilder4.check(TheoryBuilder4.java:150)
> at
> org.quicktheories.dsl.TheoryBuilder4.checkAssert(TheoryBuilder4.java:162)
> at
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest.corruptionCausesFailure(ChecksummingTransformerTest.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15313) Fix flaky - ChecksummingTransformerTest - org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
[
https://issues.apache.org/jira/browse/CASSANDRA-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West updated CASSANDRA-15313:
Bug Category: Parent values: Correctness(12982)Level 1 values: Test
Failure(12990)
Complexity: Low Hanging Fruit
Discovered By: User Report
Severity: Normal
Status: Open (was: Triage Needed)
> Fix flaky - ChecksummingTransformerTest -
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
> ---
>
> Key: CASSANDRA-15313
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15313
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Vinay Chella
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 4.0-alpha
>
> Attachments: CASSANDRA-15313-hack.patch
>
>
> During the recent runs, this test appears to be flaky.
> Example failure:
> [https://circleci.com/gh/vinaykumarchella/cassandra/459#tests/containers/94]
> corruptionCausesFailure-compression -
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
> {code:java}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
> at org.quicktheories.impl.Precursor.(Precursor.java:17)
> at
> org.quicktheories.impl.ConcreteDetachedSource.(ConcreteDetachedSource.java:8)
> at
> org.quicktheories.impl.ConcreteDetachedSource.detach(ConcreteDetachedSource.java:23)
> at org.quicktheories.generators.Retry.generate(CodePoints.java:51)
> at
> org.quicktheories.generators.Generate.lambda$intArrays$10(Generate.java:190)
> at
> org.quicktheories.generators.Generate$$Lambda$17/1847008471.generate(Unknown
> Source)
> at org.quicktheories.core.DescribingGenerator.generate(Gen.java:255)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$mix$10(Gen.java:184)
> at org.quicktheories.core.Gen$$Lambda$45/802243390.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$flatMap$5(Gen.java:93)
> at org.quicktheories.core.Gen$$Lambda$48/363509958.generate(Unknown
> Source)
> at
> org.quicktheories.dsl.TheoryBuilder4.lambda$prgnToTuple$12(TheoryBuilder4.java:188)
> at
> org.quicktheories.dsl.TheoryBuilder4$$Lambda$40/2003496028.generate(Unknown
> Source)
> at org.quicktheories.core.DescribingGenerator.generate(Gen.java:255)
> at org.quicktheories.core.FilteredGenerator.generate(Gen.java:225)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.impl.Core.generate(Core.java:150)
> at org.quicktheories.impl.Core.shrink(Core.java:103)
> at org.quicktheories.impl.Core.run(Core.java:39)
> at org.quicktheories.impl.TheoryRunner.check(TheoryRunner.java:35)
> at org.quicktheories.dsl.TheoryBuilder4.check(TheoryBuilder4.java:150)
> at
> org.quicktheories.dsl.TheoryBuilder4.checkAssert(TheoryBuilder4.java:162)
> at
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest.corruptionCausesFailure(ChecksummingTransformerTest.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15313) Fix flaky - ChecksummingTransformerTest - org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
[
https://issues.apache.org/jira/browse/CASSANDRA-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031680#comment-17031680
]
Jordan West edited comment on CASSANDRA-15313 at 2/6/20 3:40 PM:
-
[~samt] actually included the test on my behalf. Its not surprising that this
is a memory issue. Thats one motivation for
https://issues.apache.org/jira/browse/CASSANDRA-15348 which we could convert to
when ready. But in the meantime let me see what we can do to reduce the memory
usage of the test.
Interesting its also potentially finding an issue with LZ4. Have we filed a
separate JIRA for that?
was (Author: jrwest):
[~samt] actually included the test on my behalf so I can take a closer look here
> Fix flaky - ChecksummingTransformerTest -
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
> ---
>
> Key: CASSANDRA-15313
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15313
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Vinay Chella
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 4.0-alpha
>
> Attachments: CASSANDRA-15313-hack.patch
>
>
> During the recent runs, this test appears to be flaky.
> Example failure:
> [https://circleci.com/gh/vinaykumarchella/cassandra/459#tests/containers/94]
> corruptionCausesFailure-compression -
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
> {code:java}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
> at org.quicktheories.impl.Precursor.(Precursor.java:17)
> at
> org.quicktheories.impl.ConcreteDetachedSource.(ConcreteDetachedSource.java:8)
> at
> org.quicktheories.impl.ConcreteDetachedSource.detach(ConcreteDetachedSource.java:23)
> at org.quicktheories.generators.Retry.generate(CodePoints.java:51)
> at
> org.quicktheories.generators.Generate.lambda$intArrays$10(Generate.java:190)
> at
> org.quicktheories.generators.Generate$$Lambda$17/1847008471.generate(Unknown
> Source)
> at org.quicktheories.core.DescribingGenerator.generate(Gen.java:255)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$mix$10(Gen.java:184)
> at org.quicktheories.core.Gen$$Lambda$45/802243390.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$flatMap$5(Gen.java:93)
> at org.quicktheories.core.Gen$$Lambda$48/363509958.generate(Unknown
> Source)
> at
> org.quicktheories.dsl.TheoryBuilder4.lambda$prgnToTuple$12(TheoryBuilder4.java:188)
> at
> org.quicktheories.dsl.TheoryBuilder4$$Lambda$40/2003496028.generate(Unknown
> Source)
> at org.quicktheories.core.DescribingGenerator.generate(Gen.java:255)
> at org.quicktheories.core.FilteredGenerator.generate(Gen.java:225)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.impl.Core.generate(Core.java:150)
> at org.quicktheories.impl.Core.shrink(Core.java:103)
> at org.quicktheories.impl.Core.run(Core.java:39)
> at org.quicktheories.impl.TheoryRunner.check(TheoryRunner.java:35)
> at org.quicktheories.dsl.TheoryBuilder4.check(TheoryBuilder4.java:150)
> at
> org.quicktheories.dsl.TheoryBuilder4.checkAssert(TheoryBuilder4.java:162)
> at
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest.corruptionCausesFailure(ChecksummingTransformerTest.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15313) Fix flaky - ChecksummingTransformerTest - org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
[
https://issues.apache.org/jira/browse/CASSANDRA-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031680#comment-17031680
]
Jordan West commented on CASSANDRA-15313:
-
[~samt] actually included the test on my behalf so I can take a closer look here
> Fix flaky - ChecksummingTransformerTest -
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
> ---
>
> Key: CASSANDRA-15313
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15313
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Vinay Chella
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 4.0-alpha
>
> Attachments: CASSANDRA-15313-hack.patch
>
>
> During the recent runs, this test appears to be flaky.
> Example failure:
> [https://circleci.com/gh/vinaykumarchella/cassandra/459#tests/containers/94]
> corruptionCausesFailure-compression -
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
> {code:java}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
> at org.quicktheories.impl.Precursor.(Precursor.java:17)
> at
> org.quicktheories.impl.ConcreteDetachedSource.(ConcreteDetachedSource.java:8)
> at
> org.quicktheories.impl.ConcreteDetachedSource.detach(ConcreteDetachedSource.java:23)
> at org.quicktheories.generators.Retry.generate(CodePoints.java:51)
> at
> org.quicktheories.generators.Generate.lambda$intArrays$10(Generate.java:190)
> at
> org.quicktheories.generators.Generate$$Lambda$17/1847008471.generate(Unknown
> Source)
> at org.quicktheories.core.DescribingGenerator.generate(Gen.java:255)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$mix$10(Gen.java:184)
> at org.quicktheories.core.Gen$$Lambda$45/802243390.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$flatMap$5(Gen.java:93)
> at org.quicktheories.core.Gen$$Lambda$48/363509958.generate(Unknown
> Source)
> at
> org.quicktheories.dsl.TheoryBuilder4.lambda$prgnToTuple$12(TheoryBuilder4.java:188)
> at
> org.quicktheories.dsl.TheoryBuilder4$$Lambda$40/2003496028.generate(Unknown
> Source)
> at org.quicktheories.core.DescribingGenerator.generate(Gen.java:255)
> at org.quicktheories.core.FilteredGenerator.generate(Gen.java:225)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.impl.Core.generate(Core.java:150)
> at org.quicktheories.impl.Core.shrink(Core.java:103)
> at org.quicktheories.impl.Core.run(Core.java:39)
> at org.quicktheories.impl.TheoryRunner.check(TheoryRunner.java:35)
> at org.quicktheories.dsl.TheoryBuilder4.check(TheoryBuilder4.java:150)
> at
> org.quicktheories.dsl.TheoryBuilder4.checkAssert(TheoryBuilder4.java:162)
> at
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest.corruptionCausesFailure(ChecksummingTransformerTest.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15519) Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031674#comment-17031674 ] Jordan West commented on CASSANDRA-15519: - It was discussed offline that this shouldn't go into 3.0. [updated trunk branch | https://github.com/jrwest/cassandra/tree/jwest/15519-trunk] & [tests | https://circleci.com/gh/jrwest/cassandra/tree/jwest%2F15519-trunk] > Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at > runtime > - > > Key: CASSANDRA-15519 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15519 > Project: Cassandra > Issue Type: Improvement > Components: Local/Config, Messaging/Client >Reporter: Jordan West >Assignee: Jordan West >Priority: Normal > Labels: pull-request-available > Fix For: 4.0-beta > > Time Spent: 10m > Remaining Estimate: 0h > > CASSANDRA-15013 added configurable global and per endpoint limits on the > number of in flight requests, measured in bytes. During times when the > cluster is overloaded, it can be useful to change this setting without having > to restart the Cassandra process. Changing the limits should affect all > existing and new connections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15519) Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031674#comment-17031674 ] Jordan West edited comment on CASSANDRA-15519 at 2/6/20 3:28 PM: - It was discussed offline that this shouldn't go into 3.0, which is bugfix only at this time. [updated trunk branch | https://github.com/jrwest/cassandra/tree/jwest/15519-trunk] & [tests | https://circleci.com/gh/jrwest/cassandra/tree/jwest%2F15519-trunk] was (Author: jrwest): It was discussed offline that this shouldn't go into 3.0. [updated trunk branch | https://github.com/jrwest/cassandra/tree/jwest/15519-trunk] & [tests | https://circleci.com/gh/jrwest/cassandra/tree/jwest%2F15519-trunk] > Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at > runtime > - > > Key: CASSANDRA-15519 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15519 > Project: Cassandra > Issue Type: Improvement > Components: Local/Config, Messaging/Client >Reporter: Jordan West >Assignee: Jordan West >Priority: Normal > Labels: pull-request-available > Fix For: 4.0-beta > > Time Spent: 10m > Remaining Estimate: 0h > > CASSANDRA-15013 added configurable global and per endpoint limits on the > number of in flight requests, measured in bytes. During times when the > cluster is overloaded, it can be useful to change this setting without having > to restart the Cassandra process. Changing the limits should affect all > existing and new connections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15526) Fix flakey test - org.apache.cassandra.index.sasi.SASIIndexTest testConcurrentMemtableReadsAndWrites
[
https://issues.apache.org/jira/browse/CASSANDRA-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1703#comment-1703
]
Jordan West commented on CASSANDRA-15526:
-
I was able to reproduce locally and added a message to the assertion to get a
bit more insight. When the {{TrieMemIndex}} is instantiated the map has a first
and last key (it looks like a single key in this case) but size is returning 0.
{code:java}
junit.framework.AssertionFailedError: min = -7787472368264638040, max =
-7787472368264638040, count = 0 {code}
> Fix flakey test - org.apache.cassandra.index.sasi.SASIIndexTest
> testConcurrentMemtableReadsAndWrites
>
>
> Key: CASSANDRA-15526
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15526
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: David Capwell
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> {code}
> junit.framework.AssertionFailedError
> at
> org.apache.cassandra.index.sasi.utils.RangeIterator.(RangeIterator.java:46)
> at
> org.apache.cassandra.index.sasi.memory.KeyRangeIterator.(KeyRangeIterator.java:42)
> at
> org.apache.cassandra.index.sasi.memory.TrieMemIndex$ConcurrentTrie.search(TrieMemIndex.java:150)
> at
> org.apache.cassandra.index.sasi.memory.TrieMemIndex.search(TrieMemIndex.java:102)
> at
> org.apache.cassandra.index.sasi.memory.IndexMemtable.search(IndexMemtable.java:70)
> at
> org.apache.cassandra.index.sasi.conf.ColumnIndex.searchMemtable(ColumnIndex.java:138)
> at org.apache.cassandra.index.sasi.TermIterator.build(TermIterator.java:91)
> at
> org.apache.cassandra.index.sasi.plan.QueryController.getIndexes(QueryController.java:145)
> at
> org.apache.cassandra.index.sasi.plan.Operation$Builder.complete(Operation.java:434)
> at org.apache.cassandra.index.sasi.plan.QueryPlan.analyze(QueryPlan.java:57)
> at org.apache.cassandra.index.sasi.plan.QueryPlan.execute(QueryPlan.java:68)
> at
> org.apache.cassandra.index.sasi.SASIIndex.lambda$searcherFor$2(SASIIndex.java:301)
> at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:455)
> at
> org.apache.cassandra.index.sasi.SASIIndexTest.getIndexed(SASIIndexTest.java:2576)
> at
> org.apache.cassandra.index.sasi.SASIIndexTest.getPaged(SASIIndexTest.java:2537)
> at
> org.apache.cassandra.index.sasi.SASIIndexTest.testConcurrentMemtableReadsAndWrites(SASIIndexTest.java:1108)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15469) IndexInfo/ColumnIndex Refactors & Cleanup
[ https://issues.apache.org/jira/browse/CASSANDRA-15469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031021#comment-17031021 ] Jordan West commented on CASSANDRA-15469: - Thanks Dinesh! > IndexInfo/ColumnIndex Refactors & Cleanup > - > > Key: CASSANDRA-15469 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15469 > Project: Cassandra > Issue Type: Improvement > Components: Local/SSTable >Reporter: Jordan West >Assignee: Jordan West >Priority: Normal > Fix For: 4.0-alpha > > > Changes determined by audit of IndexInfo and ColumnIndex changes for > Cassandra 4.0 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15213) DecayingEstimatedHistogramReservoir Inefficiencies
[
https://issues.apache.org/jira/browse/CASSANDRA-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026770#comment-17026770
]
Jordan West edited comment on CASSANDRA-15213 at 1/30/20 3:41 PM:
--
Incorporated your change, rebased/squashed, and pushed. Thanks [~benedict].
[branch | https://github.com/jrwest/cassandra/tree/jwest/15213] [tests |
https://circleci.com/gh/jrwest/cassandra/tree/jwest%2F15213]
was (Author: jrwest):
Incorporated your change, rebased/squashed, and pushed. Thanks [~benedict].
> DecayingEstimatedHistogramReservoir Inefficiencies
> --
>
> Key: CASSANDRA-15213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15213
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Benedict Elliott Smith
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0-beta
>
>
> * {{LongAdder}} introduced to trunk consumes 9MiB of heap without user
> schemas, and this will grow significantly under contention and user schemas
> with many tables. This is because {{LongAdder}} is a very heavy class
> designed for single contended values.
> ** This can likely be improved significantly, without significant loss of
> performance in the contended case, by simply increasing the size of our
> primitive backing array and providing multiple buckets, with each thread
> picking a bucket to increment, or simply multiple backing arrays. Probably a
> better way still to do this would be to introduce some competition detection
> to the update, much like {{LongAdder}} utilises, that increases the number of
> backing arrays under competition.
> ** To save memory this approach could partition the space into chunks that
> are likely to be updated together, so that we do not need to duplicate the
> entire array under competition.
> * Similarly, binary search is costly and a measurable cost as a share of the
> new networking work (without filtering it was > 10% of the CPU used overall).
> We can compute an approximation floor(log2 n / log2 1.2) extremely cheaply,
> to save the random memory access costs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15213) DecayingEstimatedHistogramReservoir Inefficiencies
[
https://issues.apache.org/jira/browse/CASSANDRA-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026770#comment-17026770
]
Jordan West commented on CASSANDRA-15213:
-
Incorporated your change, rebased/squashed, and pushed. Thanks [~benedict].
> DecayingEstimatedHistogramReservoir Inefficiencies
> --
>
> Key: CASSANDRA-15213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15213
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Benedict Elliott Smith
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0-beta
>
>
> * {{LongAdder}} introduced to trunk consumes 9MiB of heap without user
> schemas, and this will grow significantly under contention and user schemas
> with many tables. This is because {{LongAdder}} is a very heavy class
> designed for single contended values.
> ** This can likely be improved significantly, without significant loss of
> performance in the contended case, by simply increasing the size of our
> primitive backing array and providing multiple buckets, with each thread
> picking a bucket to increment, or simply multiple backing arrays. Probably a
> better way still to do this would be to introduce some competition detection
> to the update, much like {{LongAdder}} utilises, that increases the number of
> backing arrays under competition.
> ** To save memory this approach could partition the space into chunks that
> are likely to be updated together, so that we do not need to duplicate the
> entire array under competition.
> * Similarly, binary search is costly and a measurable cost as a share of the
> new networking work (without filtering it was > 10% of the CPU used overall).
> We can compute an approximation floor(log2 n / log2 1.2) extremely cheaply,
> to save the random memory access costs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15213) DecayingEstimatedHistogramReservoir Inefficiencies
[
https://issues.apache.org/jira/browse/CASSANDRA-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026466#comment-17026466
]
Jordan West commented on CASSANDRA-15213:
-
Pushed an update to the branch that includes prime distribution
> DecayingEstimatedHistogramReservoir Inefficiencies
> --
>
> Key: CASSANDRA-15213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15213
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Benedict Elliott Smith
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0-beta
>
>
> * {{LongAdder}} introduced to trunk consumes 9MiB of heap without user
> schemas, and this will grow significantly under contention and user schemas
> with many tables. This is because {{LongAdder}} is a very heavy class
> designed for single contended values.
> ** This can likely be improved significantly, without significant loss of
> performance in the contended case, by simply increasing the size of our
> primitive backing array and providing multiple buckets, with each thread
> picking a bucket to increment, or simply multiple backing arrays. Probably a
> better way still to do this would be to introduce some competition detection
> to the update, much like {{LongAdder}} utilises, that increases the number of
> backing arrays under competition.
> ** To save memory this approach could partition the space into chunks that
> are likely to be updated together, so that we do not need to duplicate the
> entire array under competition.
> * Similarly, binary search is costly and a measurable cost as a share of the
> new networking work (without filtering it was > 10% of the CPU used overall).
> We can compute an approximation floor(log2 n / log2 1.2) extremely cheaply,
> to save the random memory access costs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15533) Don't allocate unneeded MergeIterator in OnDiskToken#iterator
[ https://issues.apache.org/jira/browse/CASSANDRA-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jordan West updated CASSANDRA-15533: Test and Documentation Plan: existing test suite Status: Patch Available (was: Open) [branch|https://github.com/jrwest/cassandra/tree/jwest/15533] [tests|https://circleci.com/gh/jrwest/cassandra/tree/jwest%2F15533] > Don't allocate unneeded MergeIterator in OnDiskToken#iterator > -- > > Key: CASSANDRA-15533 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15533 > Project: Cassandra > Issue Type: Improvement > Components: Feature/SASI >Reporter: Jordan West >Assignee: Jordan West >Priority: Normal > Fix For: 4.0-rc > > > When reviewing CASSANDRA-15392 it became apparent that the MergeIterator > allocated by OnDiskToken#iterator is rarely necessary and so we should avoid > allocating one when not needed and skip the MergeIterator pool when needed > because its unlikely to be sized correctly. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15533) Don't allocate unneeded MergeIterator in OnDiskToken#iterator
[ https://issues.apache.org/jira/browse/CASSANDRA-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jordan West updated CASSANDRA-15533: Change Category: Performance Complexity: Low Hanging Fruit Fix Version/s: 4.0-rc Status: Open (was: Triage Needed) > Don't allocate unneeded MergeIterator in OnDiskToken#iterator > -- > > Key: CASSANDRA-15533 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15533 > Project: Cassandra > Issue Type: Improvement > Components: Feature/SASI >Reporter: Jordan West >Assignee: Jordan West >Priority: Normal > Fix For: 4.0-rc > > > When reviewing CASSANDRA-15392 it became apparent that the MergeIterator > allocated by OnDiskToken#iterator is rarely necessary and so we should avoid > allocating one when not needed and skip the MergeIterator pool when needed > because its unlikely to be sized correctly. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15533) Don't allocate unneeded MergeIterator in OnDiskToken#iterator
Jordan West created CASSANDRA-15533: --- Summary: Don't allocate unneeded MergeIterator in OnDiskToken#iterator Key: CASSANDRA-15533 URL: https://issues.apache.org/jira/browse/CASSANDRA-15533 Project: Cassandra Issue Type: Improvement Components: Feature/SASI Reporter: Jordan West Assignee: Jordan West When reviewing CASSANDRA-15392 it became apparent that the MergeIterator allocated by OnDiskToken#iterator is rarely necessary and so we should avoid allocating one when not needed and skip the MergeIterator pool when needed because its unlikely to be sized correctly. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15519) Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jordan West updated CASSANDRA-15519: Reviewers: Sumanth Pasupuleti (was: Jordan West, Sumanth Pasupuleti) > Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at > runtime > - > > Key: CASSANDRA-15519 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15519 > Project: Cassandra > Issue Type: Improvement > Components: Local/Config, Messaging/Client >Reporter: Jordan West >Assignee: Jordan West >Priority: Normal > Labels: pull-request-available > Fix For: 4.0-beta > > Time Spent: 10m > Remaining Estimate: 0h > > CASSANDRA-15013 added configurable global and per endpoint limits on the > number of in flight requests, measured in bytes. During times when the > cluster is overloaded, it can be useful to change this setting without having > to restart the Cassandra process. Changing the limits should affect all > existing and new connections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15519) Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jordan West updated CASSANDRA-15519: Reviewers: Sumanth Pasupuleti, Jordan West (was: Jordan West, Sumanth Pasupuleti) Sumanth Pasupuleti, Jordan West Status: Review In Progress (was: Patch Available) > Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at > runtime > - > > Key: CASSANDRA-15519 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15519 > Project: Cassandra > Issue Type: Improvement > Components: Local/Config, Messaging/Client >Reporter: Jordan West >Assignee: Jordan West >Priority: Normal > Labels: pull-request-available > Fix For: 4.0-beta > > Time Spent: 10m > Remaining Estimate: 0h > > CASSANDRA-15013 added configurable global and per endpoint limits on the > number of in flight requests, measured in bytes. During times when the > cluster is overloaded, it can be useful to change this setting without having > to restart the Cassandra process. Changing the limits should affect all > existing and new connections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15470) Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB & Bytes
[
https://issues.apache.org/jira/browse/CASSANDRA-15470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024885#comment-17024885
]
Jordan West commented on CASSANDRA-15470:
-
Thanks Dinesh. I like the improvements to {{ByteUnit}}. I agree that a more
wholesale review of {{DatabaseDescriptor}} would be good but is best saved for
a separate ticket and at a different time in the release cycle.
> Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB
> & Bytes
> -
>
> Key: CASSANDRA-15470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15470
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Config
>Reporter: Jordan West
>Assignee: Mallika Kulkarni
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-rc
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{DatabaseDescriptor}} has several functions that convert between user
> supplied sizes in KB/MB and bytes. These are implemented without much
> consistency and, while unlikely, several have the potential to overflow since
> validation on the input is missing. Meanwhile, some widen the number to a
> long correctly. Options include: widening in all places or simply doing
> better validation on start up — currently only the lower bound of the valid
> range is checked for many of these fields.
> List of Affected {{DatabaseDescriptor}} Methods:
> * {{getColumnIndexSize}}
> * {{getColumnIndexCacheSize}}
> * {{getBatchSizeWarnThreshold}}
> * {{getNativeTransportFrameBlockSize}}
> * {{getRepairSessionSpaceInMegabytes}}
> * {{getNativeTransportMaxFrameSize}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15470) Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB & Bytes
[
https://issues.apache.org/jira/browse/CASSANDRA-15470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024885#comment-17024885
]
Jordan West edited comment on CASSANDRA-15470 at 1/28/20 5:59 AM:
--
Thanks [~djoshi]. I like the improvements to {{ByteUnit}}. I agree that a more
wholesale review of {{DatabaseDescriptor}} would be good but is best saved for
a separate ticket and at a different time in the release cycle.
+1
was (Author: jrwest):
Thanks [~djoshi]. I like the improvements to {{ByteUnit}}. I agree that a more
wholesale review of {{DatabaseDescriptor}} would be good but is best saved for
a separate ticket and at a different time in the release cycle.
> Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB
> & Bytes
> -
>
> Key: CASSANDRA-15470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15470
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Config
>Reporter: Jordan West
>Assignee: Mallika Kulkarni
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-rc
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{DatabaseDescriptor}} has several functions that convert between user
> supplied sizes in KB/MB and bytes. These are implemented without much
> consistency and, while unlikely, several have the potential to overflow since
> validation on the input is missing. Meanwhile, some widen the number to a
> long correctly. Options include: widening in all places or simply doing
> better validation on start up — currently only the lower bound of the valid
> range is checked for many of these fields.
> List of Affected {{DatabaseDescriptor}} Methods:
> * {{getColumnIndexSize}}
> * {{getColumnIndexCacheSize}}
> * {{getBatchSizeWarnThreshold}}
> * {{getNativeTransportFrameBlockSize}}
> * {{getRepairSessionSpaceInMegabytes}}
> * {{getNativeTransportMaxFrameSize}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15469) IndexInfo/ColumnIndex Refactors & Cleanup
[ https://issues.apache.org/jira/browse/CASSANDRA-15469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024887#comment-17024887 ] Jordan West commented on CASSANDRA-15469: - Pushed an update to the branch with the discussed changes > IndexInfo/ColumnIndex Refactors & Cleanup > - > > Key: CASSANDRA-15469 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15469 > Project: Cassandra > Issue Type: Improvement > Components: Local/SSTable >Reporter: Jordan West >Assignee: Jordan West >Priority: Normal > Fix For: 4.0-beta > > > Changes determined by audit of IndexInfo and ColumnIndex changes for > Cassandra 4.0 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15470) Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB & Bytes
[
https://issues.apache.org/jira/browse/CASSANDRA-15470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024885#comment-17024885
]
Jordan West edited comment on CASSANDRA-15470 at 1/28/20 5:59 AM:
--
Thanks [~djoshi]. I like the improvements to {{ByteUnit}}. I agree that a more
wholesale review of {{DatabaseDescriptor}} would be good but is best saved for
a separate ticket and at a different time in the release cycle.
was (Author: jrwest):
Thanks Dinesh. I like the improvements to {{ByteUnit}}. I agree that a more
wholesale review of {{DatabaseDescriptor}} would be good but is best saved for
a separate ticket and at a different time in the release cycle.
> Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB
> & Bytes
> -
>
> Key: CASSANDRA-15470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15470
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Config
>Reporter: Jordan West
>Assignee: Mallika Kulkarni
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-rc
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{DatabaseDescriptor}} has several functions that convert between user
> supplied sizes in KB/MB and bytes. These are implemented without much
> consistency and, while unlikely, several have the potential to overflow since
> validation on the input is missing. Meanwhile, some widen the number to a
> long correctly. Options include: widening in all places or simply doing
> better validation on start up — currently only the lower bound of the valid
> range is checked for many of these fields.
> List of Affected {{DatabaseDescriptor}} Methods:
> * {{getColumnIndexSize}}
> * {{getColumnIndexCacheSize}}
> * {{getBatchSizeWarnThreshold}}
> * {{getNativeTransportFrameBlockSize}}
> * {{getRepairSessionSpaceInMegabytes}}
> * {{getNativeTransportMaxFrameSize}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15519) Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at runtime
[
https://issues.apache.org/jira/browse/CASSANDRA-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024874#comment-17024874
]
Jordan West edited comment on CASSANDRA-15519 at 1/28/20 5:20 AM:
--
[~clohfink] and I spoke off JIRA and moving things into {{StorageServiceMBean}}
works well and cleans things up nicely as an added bonus. Branch updated.
was (Author: jrwest):
[~clohfink] and I spoke offline and moving things into {{StorageServiceMBean}}
works well and cleans things up nicely as an added bonus. Branch updated.
> Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at
> runtime
> -
>
> Key: CASSANDRA-15519
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15519
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config, Messaging/Client
>Reporter: Jordan West
>Assignee: Jordan West
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-beta
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> CASSANDRA-15013 added configurable global and per endpoint limits on the
> number of in flight requests, measured in bytes. During times when the
> cluster is overloaded, it can be useful to change this setting without having
> to restart the Cassandra process. Changing the limits should affect all
> existing and new connections.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15519) Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at runtime
[
https://issues.apache.org/jira/browse/CASSANDRA-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024874#comment-17024874
]
Jordan West commented on CASSANDRA-15519:
-
[~clohfink] and I spoke offline and moving things into {{StorageServiceMBean}}
works well and cleans things up nicely as an added bonus. Branch updated.
> Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at
> runtime
> -
>
> Key: CASSANDRA-15519
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15519
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config, Messaging/Client
>Reporter: Jordan West
>Assignee: Jordan West
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-beta
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> CASSANDRA-15013 added configurable global and per endpoint limits on the
> number of in flight requests, measured in bytes. During times when the
> cluster is overloaded, it can be useful to change this setting without having
> to restart the Cassandra process. Changing the limits should affect all
> existing and new connections.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15470) Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB & Bytes
[
https://issues.apache.org/jira/browse/CASSANDRA-15470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024712#comment-17024712
]
Jordan West commented on CASSANDRA-15470:
-
[~djoshi] moving the {{ByteUnit}} changes out to a separate ticket SGTM. Can
you say a bit more about what test failed and why? I missed that. Thanks.
> Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB
> & Bytes
> -
>
> Key: CASSANDRA-15470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15470
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Config
>Reporter: Jordan West
>Assignee: Mallika Kulkarni
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-rc
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{DatabaseDescriptor}} has several functions that convert between user
> supplied sizes in KB/MB and bytes. These are implemented without much
> consistency and, while unlikely, several have the potential to overflow since
> validation on the input is missing. Meanwhile, some widen the number to a
> long correctly. Options include: widening in all places or simply doing
> better validation on start up — currently only the lower bound of the valid
> range is checked for many of these fields.
> List of Affected {{DatabaseDescriptor}} Methods:
> * {{getColumnIndexSize}}
> * {{getColumnIndexCacheSize}}
> * {{getBatchSizeWarnThreshold}}
> * {{getNativeTransportFrameBlockSize}}
> * {{getRepairSessionSpaceInMegabytes}}
> * {{getNativeTransportMaxFrameSize}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15519) Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at runtime
[
https://issues.apache.org/jira/browse/CASSANDRA-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024563#comment-17024563
]
Jordan West edited comment on CASSANDRA-15519 at 1/27/20 6:53 PM:
--
Totally understand the benefit of having things near each other but
unfortunately here the feature isn't implemented near that area (the CQL layer
is a clear dividing line between transport and execution). The feature itself
(the queue management) is per {{Server}} as implemented. Is there another
middle ground or recommended place that isn't {{StorageProxyMBean}} we could
use?
was (Author: jrwest):
Totally understand the benefit of having things near each other but
unfortunately here the feature isn't implemented near that area (the CQL layer
is a clear dividing line between transport and execution). The feature itself
(the queue management) is per Server as implemented. Is there another middle
ground or recommended place that isn't {{StorageProxyMBean}} we could use?
> Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at
> runtime
> -
>
> Key: CASSANDRA-15519
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15519
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config, Messaging/Client
>Reporter: Jordan West
>Assignee: Jordan West
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-beta
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> CASSANDRA-15013 added configurable global and per endpoint limits on the
> number of in flight requests, measured in bytes. During times when the
> cluster is overloaded, it can be useful to change this setting without having
> to restart the Cassandra process. Changing the limits should affect all
> existing and new connections.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15519) Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at runtime
[
https://issues.apache.org/jira/browse/CASSANDRA-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024563#comment-17024563
]
Jordan West commented on CASSANDRA-15519:
-
Totally understand the benefit of having things near each other but
unfortunately here the feature isn't implemented near that area (the CQL layer
is a clear dividing line between transport and execution). The feature itself
(the queue management) is per Server as implemented. Is there another middle
ground or recommended place that isn't {{StorageProxyMBean}} we could use?
> Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at
> runtime
> -
>
> Key: CASSANDRA-15519
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15519
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config, Messaging/Client
>Reporter: Jordan West
>Assignee: Jordan West
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-beta
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> CASSANDRA-15013 added configurable global and per endpoint limits on the
> number of in flight requests, measured in bytes. During times when the
> cluster is overloaded, it can be useful to change this setting without having
> to restart the Cassandra process. Changing the limits should affect all
> existing and new connections.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15470) Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB & Bytes
[
https://issues.apache.org/jira/browse/CASSANDRA-15470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024554#comment-17024554
]
Jordan West commented on CASSANDRA-15470:
-
I'm generally fine with either the original patch I reviewed or the newer
version. I'm not sure it was necessary to do the renaming here (when other
functions don't follow that convention yet) and I think {{ByteUnit}} could be a
little more developed.
> Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB
> & Bytes
> -
>
> Key: CASSANDRA-15470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15470
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Config
>Reporter: Jordan West
>Assignee: Mallika Kulkarni
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-rc
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{DatabaseDescriptor}} has several functions that convert between user
> supplied sizes in KB/MB and bytes. These are implemented without much
> consistency and, while unlikely, several have the potential to overflow since
> validation on the input is missing. Meanwhile, some widen the number to a
> long correctly. Options include: widening in all places or simply doing
> better validation on start up — currently only the lower bound of the valid
> range is checked for many of these fields.
> List of Affected {{DatabaseDescriptor}} Methods:
> * {{getColumnIndexSize}}
> * {{getColumnIndexCacheSize}}
> * {{getBatchSizeWarnThreshold}}
> * {{getNativeTransportFrameBlockSize}}
> * {{getRepairSessionSpaceInMegabytes}}
> * {{getNativeTransportMaxFrameSize}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15469) IndexInfo/ColumnIndex Refactors & Cleanup
[
https://issues.apache.org/jira/browse/CASSANDRA-15469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022698#comment-17022698
]
Jordan West commented on CASSANDRA-15469:
-
Good catch [~djoshi]. I think these could be addressed storing the cache size
in a field and updating it on every call to {{reset()}}. The buffer allocation
issue could be addressed by reallocating when a difference in the size of the
buffer and the expected size given the current field value is detected. WDTY?
One issue I see tangentially related to this is that updating the value doesn't
affect the existing cached indexes. I think its safe to assume that an operator
can drop the cache if needed but I'd like to check w/ [~clohfink] here if
something more heavy handed is desired re: existing cached items when updating
the cache size property.
> IndexInfo/ColumnIndex Refactors & Cleanup
> -
>
> Key: CASSANDRA-15469
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15469
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/SSTable
>Reporter: Jordan West
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0-beta
>
>
> Changes determined by audit of IndexInfo and ColumnIndex changes for
> Cassandra 4.0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15519) Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022647#comment-17022647 ] Jordan West commented on CASSANDRA-15519: - Fixed the [tests|https://circleci.com/gh/jrwest/cassandra/tree/jwest%2F15519] > Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at > runtime > - > > Key: CASSANDRA-15519 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15519 > Project: Cassandra > Issue Type: Improvement > Components: Local/Config, Messaging/Client >Reporter: Jordan West >Assignee: Jordan West >Priority: Normal > Labels: pull-request-available > Fix For: 4.0-beta > > Time Spent: 10m > Remaining Estimate: 0h > > CASSANDRA-15013 added configurable global and per endpoint limits on the > number of in flight requests, measured in bytes. During times when the > cluster is overloaded, it can be useful to change this setting without having > to restart the Cassandra process. Changing the limits should affect all > existing and new connections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15519) Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022637#comment-17022637 ] Jordan West commented on CASSANDRA-15519: - [~clohfink] thanks for the input. Could you help me understand a bit better the value in moving the methods (I can certainly make them more meaningful in either place)? The reason I ask is because from my (limited) perspective its pretty much the same difficultly to change the values regardless of the MBean they live in and discovery is a matter of documentation (which I can add as part of this patch). I'm open to making the change if your experience tells you it will be much more valuable – my hesitation is that it exposes either Server to StorageProxy or visa-versa when currently there is a pretty clean delineation of layers there. > Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at > runtime > - > > Key: CASSANDRA-15519 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15519 > Project: Cassandra > Issue Type: Improvement > Components: Local/Config, Messaging/Client >Reporter: Jordan West >Assignee: Jordan West >Priority: Normal > Labels: pull-request-available > Fix For: 4.0-beta > > Time Spent: 10m > Remaining Estimate: 0h > > CASSANDRA-15013 added configurable global and per endpoint limits on the > number of in flight requests, measured in bytes. During times when the > cluster is overloaded, it can be useful to change this setting without having > to restart the Cassandra process. Changing the limits should affect all > existing and new connections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15519) Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022232#comment-17022232 ] Jordan West commented on CASSANDRA-15519: - Also looks like I broke a few unit tests w/ the MBean registration. Will fix. > Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at > runtime > - > > Key: CASSANDRA-15519 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15519 > Project: Cassandra > Issue Type: Improvement > Components: Local/Config, Messaging/Client >Reporter: Jordan West >Assignee: Jordan West >Priority: Normal > Labels: pull-request-available > Fix For: 4.0-beta > > Time Spent: 10m > Remaining Estimate: 0h > > CASSANDRA-15013 added configurable global and per endpoint limits on the > number of in flight requests, measured in bytes. During times when the > cluster is overloaded, it can be useful to change this setting without having > to restart the Cassandra process. Changing the limits should affect all > existing and new connections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15519) Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jordan West updated CASSANDRA-15519: Test and Documentation Plan: Added unit test and manually tested JMX modifications via jconsole Status: Patch Available (was: Open) A couple notes for reviewers: * I used the BiConsumer (remover) pattern in EndpointPayloadTracker to make the code more testable but if there is concern about the memory overhead I can change it * Two concurrent calls to setEndpointLimit could leave the node in a state where some endpoints have one limit and some have the other. This would require human’s to make JMX calls concurrently so I didn’t see much value in adding more concurrency control (especially since the fix is to just set the limit one more time w/ human coordination). Open to revisiting though. > Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at > runtime > - > > Key: CASSANDRA-15519 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15519 > Project: Cassandra > Issue Type: Improvement > Components: Local/Config, Messaging/Client >Reporter: Jordan West >Assignee: Jordan West >Priority: Normal > Labels: pull-request-available > Fix For: 4.0-beta > > Time Spent: 10m > Remaining Estimate: 0h > > CASSANDRA-15013 added configurable global and per endpoint limits on the > number of in flight requests, measured in bytes. During times when the > cluster is overloaded, it can be useful to change this setting without having > to restart the Cassandra process. Changing the limits should affect all > existing and new connections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15519) Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jordan West updated CASSANDRA-15519: Change Category: Operability Complexity: Normal Fix Version/s: 4.0-beta Status: Open (was: Triage Needed) > Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at > runtime > - > > Key: CASSANDRA-15519 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15519 > Project: Cassandra > Issue Type: Improvement > Components: Local/Config, Messaging/Client >Reporter: Jordan West >Assignee: Jordan West >Priority: Normal > Fix For: 4.0-beta > > > CASSANDRA-15013 added configurable global and per endpoint limits on the > number of in flight requests, measured in bytes. During times when the > cluster is overloaded, it can be useful to change this setting without having > to restart the Cassandra process. Changing the limits should affect all > existing and new connections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15519) Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at runtime
Jordan West created CASSANDRA-15519: --- Summary: Make native_transport_max_concurrent_requests_in_bytes(_per_ip) changeable at runtime Key: CASSANDRA-15519 URL: https://issues.apache.org/jira/browse/CASSANDRA-15519 Project: Cassandra Issue Type: Improvement Components: Local/Config, Messaging/Client Reporter: Jordan West Assignee: Jordan West CASSANDRA-15013 added configurable global and per endpoint limits on the number of in flight requests, measured in bytes. During times when the cluster is overloaded, it can be useful to change this setting without having to restart the Cassandra process. Changing the limits should affect all existing and new connections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15392) Pool Merge Iterators
[
https://issues.apache.org/jira/browse/CASSANDRA-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022051#comment-17022051
]
Jordan West commented on CASSANDRA-15392:
-
{quote} As this relates to SASI, this wouldn’t be a regression right? I'd guess
we're already wasting TrivialOneToOne iterators?
{quote}
Correct. This wouldn't break SASI but it would waste TrivialOneToOne's most of
the time and rarely it would waste a ManyToOne that is oversized for its needs.
I haven't looked at FQL but for SASI I think its easy enough to just skip using
MergeIterators when they aren't needed / skip the pool when they are – happy to
submit a patch if its helpful.
> Pool Merge Iterators
>
>
> Key: CASSANDRA-15392
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15392
> Project: Cassandra
> Issue Type: Sub-task
> Components: Local/Compaction
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 4.0
>
>
> By pooling merge iterators, instead of creating new ones each time we need
> them, we can reduce garbage on the compaction and read paths under relevant
> workloads by ~4% in many cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15470) Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB & Bytes
[
https://issues.apache.org/jira/browse/CASSANDRA-15470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019736#comment-17019736
]
Jordan West commented on CASSANDRA-15470:
-
[~mallika] ah right thank you. and thanks for the corrections. The patch LGTM.
I ran the new test locally and have started a circle run
[here|https://circleci.com/gh/jrwest/cassandra/tree/CASSANDRA-15470]. [~djoshi]
can you review as well? Thanks
> Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB
> & Bytes
> -
>
> Key: CASSANDRA-15470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15470
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Config
>Reporter: Jordan West
>Assignee: Mallika Kulkarni
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-rc
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{DatabaseDescriptor}} has several functions that convert between user
> supplied sizes in KB/MB and bytes. These are implemented without much
> consistency and, while unlikely, several have the potential to overflow since
> validation on the input is missing. Meanwhile, some widen the number to a
> long correctly. Options include: widening in all places or simply doing
> better validation on start up — currently only the lower bound of the valid
> range is checked for many of these fields.
> List of Affected {{DatabaseDescriptor}} Methods:
> * {{getColumnIndexSize}}
> * {{getColumnIndexCacheSize}}
> * {{getBatchSizeWarnThreshold}}
> * {{getNativeTransportFrameBlockSize}}
> * {{getRepairSessionSpaceInMegabytes}}
> * {{getNativeTransportMaxFrameSize}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15470) Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB & Bytes
[
https://issues.apache.org/jira/browse/CASSANDRA-15470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018689#comment-17018689
]
Jordan West commented on CASSANDRA-15470:
-
* Thank you for adding the validation in the {{set*}} functions. The condition
in {{setColumnIndexSize}} is using bitwise OR instead of logical OR.
* The error message for {{conf.native_transport_max_frame_size_in_mb}} uses
KiB as the unit. Also, is there a reason you picked 2 GiB as a frame size
limit? On one hand it seems like a reasonable limit, on the other, within the
scope of this ticket (preventing overflows when converting to bytes) I don’t
know if we want to artificially limit it.
> Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB
> & Bytes
> -
>
> Key: CASSANDRA-15470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15470
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Config
>Reporter: Jordan West
>Assignee: Mallika Kulkarni
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-rc
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{DatabaseDescriptor}} has several functions that convert between user
> supplied sizes in KB/MB and bytes. These are implemented without much
> consistency and, while unlikely, several have the potential to overflow since
> validation on the input is missing. Meanwhile, some widen the number to a
> long correctly. Options include: widening in all places or simply doing
> better validation on start up — currently only the lower bound of the valid
> range is checked for many of these fields.
> List of Affected {{DatabaseDescriptor}} Methods:
> * {{getColumnIndexSize}}
> * {{getColumnIndexCacheSize}}
> * {{getBatchSizeWarnThreshold}}
> * {{getNativeTransportFrameBlockSize}}
> * {{getRepairSessionSpaceInMegabytes}}
> * {{getNativeTransportMaxFrameSize}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14781) Log message when mutation passed to CommitLog#add(Mutation) is too large is not descriptive enough
[ https://issues.apache.org/jira/browse/CASSANDRA-14781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018144#comment-17018144 ] Jordan West commented on CASSANDRA-14781: - Thanks [~n.v.harikrishna]. I think the duplicate code is probably simpler than making things more nuanced and thanks for testing the sorting performance. The code looks ok me and I am ok w/ the separate ticket approach so that there is some improvements for operators at a minimum but I would like to get [~aleksey]'s input as well. > Log message when mutation passed to CommitLog#add(Mutation) is too large is > not descriptive enough > -- > > Key: CASSANDRA-14781 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14781 > Project: Cassandra > Issue Type: Bug > Components: Consistency/Hints, Local/Commit Log, Messaging/Client >Reporter: Jordan West >Assignee: Tom Petracca >Priority: Normal > Labels: protocolv5 > Fix For: 4.0-beta > > Attachments: CASSANDRA-14781.patch, CASSANDRA-14781_3.0.patch, > CASSANDRA-14781_3.11.patch > > > When hitting > [https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/commitlog/CommitLog.java#L256-L257], > the log message produced does not help the operator track down what data is > being written. At a minimum the keyspace and cfIds involved would be useful > (and are available) – more detail might not be reasonable to include. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15470) Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB & Bytes
[
https://issues.apache.org/jira/browse/CASSANDRA-15470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West updated CASSANDRA-15470:
Status: Changes Suggested (was: Review In Progress)
> Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB
> & Bytes
> -
>
> Key: CASSANDRA-15470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15470
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Config
>Reporter: Jordan West
>Assignee: Mallika Kulkarni
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-rc
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{DatabaseDescriptor}} has several functions that convert between user
> supplied sizes in KB/MB and bytes. These are implemented without much
> consistency and, while unlikely, several have the potential to overflow since
> validation on the input is missing. Meanwhile, some widen the number to a
> long correctly. Options include: widening in all places or simply doing
> better validation on start up — currently only the lower bound of the valid
> range is checked for many of these fields.
> List of Affected {{DatabaseDescriptor}} Methods:
> * {{getColumnIndexSize}}
> * {{getColumnIndexCacheSize}}
> * {{getBatchSizeWarnThreshold}}
> * {{getNativeTransportFrameBlockSize}}
> * {{getRepairSessionSpaceInMegabytes}}
> * {{getNativeTransportMaxFrameSize}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15470) Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB & Bytes
[
https://issues.apache.org/jira/browse/CASSANDRA-15470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017345#comment-17017345
]
Jordan West commented on CASSANDRA-15470:
-
Thanks [~mallika]. Comments below:
- Thank you for adding a test!
- For the {{_mb}} fields the error messages should read 2TiB instead of GiB.
- Instead of having separate if/else if clauses for the min and max, consider
combining conditions (e.g. {{conf.column_index_size_in_kb <= 0 ||
conf.column_index_size_in_kb >= 2 * 1024 * 1024}}) and making the log message
read something like “[field] must be a positive value less than 2GiB, but was”
- Consider making {{2 * 1024 * 1024}} a constant.
> Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB
> & Bytes
> -
>
> Key: CASSANDRA-15470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15470
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Config
>Reporter: Jordan West
>Assignee: Mallika Kulkarni
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-rc
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{DatabaseDescriptor}} has several functions that convert between user
> supplied sizes in KB/MB and bytes. These are implemented without much
> consistency and, while unlikely, several have the potential to overflow since
> validation on the input is missing. Meanwhile, some widen the number to a
> long correctly. Options include: widening in all places or simply doing
> better validation on start up — currently only the lower bound of the valid
> range is checked for many of these fields.
> List of Affected {{DatabaseDescriptor}} Methods:
> * {{getColumnIndexSize}}
> * {{getColumnIndexCacheSize}}
> * {{getBatchSizeWarnThreshold}}
> * {{getNativeTransportFrameBlockSize}}
> * {{getRepairSessionSpaceInMegabytes}}
> * {{getNativeTransportMaxFrameSize}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15470) Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB & Bytes
[
https://issues.apache.org/jira/browse/CASSANDRA-15470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West updated CASSANDRA-15470:
Reviewers: Jordan West, Jordan West (was: Jordan West)
Jordan West, Jordan West
Status: Review In Progress (was: Patch Available)
> Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB
> & Bytes
> -
>
> Key: CASSANDRA-15470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15470
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Config
>Reporter: Jordan West
>Assignee: Mallika Kulkarni
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-rc
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{DatabaseDescriptor}} has several functions that convert between user
> supplied sizes in KB/MB and bytes. These are implemented without much
> consistency and, while unlikely, several have the potential to overflow since
> validation on the input is missing. Meanwhile, some widen the number to a
> long correctly. Options include: widening in all places or simply doing
> better validation on start up — currently only the lower bound of the valid
> range is checked for many of these fields.
> List of Affected {{DatabaseDescriptor}} Methods:
> * {{getColumnIndexSize}}
> * {{getColumnIndexCacheSize}}
> * {{getBatchSizeWarnThreshold}}
> * {{getNativeTransportFrameBlockSize}}
> * {{getRepairSessionSpaceInMegabytes}}
> * {{getNativeTransportMaxFrameSize}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15213) DecayingEstimatedHistogramReservoir Inefficiencies
[
https://issues.apache.org/jira/browse/CASSANDRA-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016076#comment-17016076
]
Jordan West commented on CASSANDRA-15213:
-
Sounds good. Thanks for the test / proof.
> DecayingEstimatedHistogramReservoir Inefficiencies
> --
>
> Key: CASSANDRA-15213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15213
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Benedict Elliott Smith
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0-beta
>
>
> * {{LongAdder}} introduced to trunk consumes 9MiB of heap without user
> schemas, and this will grow significantly under contention and user schemas
> with many tables. This is because {{LongAdder}} is a very heavy class
> designed for single contended values.
> ** This can likely be improved significantly, without significant loss of
> performance in the contended case, by simply increasing the size of our
> primitive backing array and providing multiple buckets, with each thread
> picking a bucket to increment, or simply multiple backing arrays. Probably a
> better way still to do this would be to introduce some competition detection
> to the update, much like {{LongAdder}} utilises, that increases the number of
> backing arrays under competition.
> ** To save memory this approach could partition the space into chunks that
> are likely to be updated together, so that we do not need to duplicate the
> entire array under competition.
> * Similarly, binary search is costly and a measurable cost as a share of the
> new networking work (without filtering it was > 10% of the CPU used overall).
> We can compute an approximation floor(log2 n / log2 1.2) extremely cheaply,
> to save the random memory access costs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15213) DecayingEstimatedHistogramReservoir Inefficiencies
[
https://issues.apache.org/jira/browse/CASSANDRA-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016055#comment-17016055
]
Jordan West edited comment on CASSANDRA-15213 at 1/15/20 3:15 PM:
--
Thanks. I'll start exploring that approach. I implemented a version using
{{Integer.reverse}} (which distributed well) but didn't find an approach using
it that didn't involve an extra read/load (was looking for something more along
the lines of a simple calculation like this). Will report back with my testing
results / findings.
was (Author: jrwest):
Thanks. I'll start exploring that approach. I implemented a version using
{{Integer.reverse}} (which distributed well) but didn't find an approach using
it that didn't involve an extra read/load (was looking for something more along
the lines of a simple calculation like this). Will report back with my testing
results / findings.
EDIT: 17 divides several potential custom bucket sizes included 102, 170, and
204. To satisfy the last requirement I think we need to pick a prime such that
prime * 2 > max bucket count.
> DecayingEstimatedHistogramReservoir Inefficiencies
> --
>
> Key: CASSANDRA-15213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15213
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Benedict Elliott Smith
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0-beta
>
>
> * {{LongAdder}} introduced to trunk consumes 9MiB of heap without user
> schemas, and this will grow significantly under contention and user schemas
> with many tables. This is because {{LongAdder}} is a very heavy class
> designed for single contended values.
> ** This can likely be improved significantly, without significant loss of
> performance in the contended case, by simply increasing the size of our
> primitive backing array and providing multiple buckets, with each thread
> picking a bucket to increment, or simply multiple backing arrays. Probably a
> better way still to do this would be to introduce some competition detection
> to the update, much like {{LongAdder}} utilises, that increases the number of
> backing arrays under competition.
> ** To save memory this approach could partition the space into chunks that
> are likely to be updated together, so that we do not need to duplicate the
> entire array under competition.
> * Similarly, binary search is costly and a measurable cost as a share of the
> new networking work (without filtering it was > 10% of the CPU used overall).
> We can compute an approximation floor(log2 n / log2 1.2) extremely cheaply,
> to save the random memory access costs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
