[jira] [Commented] (CASSANDRA-16036) Add flag to disable chunk cache and disable by default
[
https://issues.apache.org/jira/browse/CASSANDRA-16036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17173546#comment-17173546
]
David Capwell commented on CASSANDRA-16036:
---
Fair point. I am spinning up two clusters, with and without CC, and can run a
few benchmarks Monday.
The current test used was a write heavy clustering key test with slices on
clustering, but mostly looked at the selects for this patch. Given the logic I
see the next test I will run is a 100% read test with pre-populated data.
> Add flag to disable chunk cache and disable by default
> --
>
> Key: CASSANDRA-16036
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16036
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: Histogram-11.png
>
>
> Chunk cache is enabled by default and doesn’t have a flag to disable without
> impacting networking. In performance testing 4.0 against 3.0 I found that
> reads were slower in 4.0 and after profiling found that the ChunkCache was
> partially to blame; after disabling the chunk cache, read performance had
> improved.
> {code}
> 40_w_cc-selects.hdr
> #[Mean= 11.50063, StdDeviation = 13.44014]
> #[Max =482.41254, Total count= 316477]
> #[Buckets = 25, SubBuckets = 262144]
> 40_wo_cc-selects.hdr
> #[Mean= 9.82115, StdDeviation = 10.14270]
> #[Max =522.36493, Total count= 317444]
> #[Buckets = 25, SubBuckets = 262144]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16036) Add flag to disable chunk cache and disable by default
[
https://issues.apache.org/jira/browse/CASSANDRA-16036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jon Meredith updated CASSANDRA-16036:
-
Reviewers: Jon Meredith, Jon Meredith (was: Jon Meredith)
Jon Meredith, Jon Meredith
Status: Review In Progress (was: Patch Available)
I took a first pass through. I definitely think we should be able to disable
the {{ChunkCache}} without impacting the \{{BufferPool}} and seem like they
should be two separate configuration entries to me.
The code changes look good too.
The only thing that needs more discussion is the change to make the ChunkCache
be disabled by default. A single benchmark isn't enough evidence to convince
me.
We could either leave the default as enabled (in which case I think it's
already good to go once CI is happy), or if we are able to collect a few more
benchmarks (read-heavy, write-heavy, compaction-heavy, hot-key) to justify
disabling then we should do so.
> Add flag to disable chunk cache and disable by default
> --
>
> Key: CASSANDRA-16036
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16036
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: Histogram-11.png
>
>
> Chunk cache is enabled by default and doesn’t have a flag to disable without
> impacting networking. In performance testing 4.0 against 3.0 I found that
> reads were slower in 4.0 and after profiling found that the ChunkCache was
> partially to blame; after disabling the chunk cache, read performance had
> improved.
> {code}
> 40_w_cc-selects.hdr
> #[Mean= 11.50063, StdDeviation = 13.44014]
> #[Max =482.41254, Total count= 316477]
> #[Buckets = 25, SubBuckets = 262144]
> 40_wo_cc-selects.hdr
> #[Mean= 9.82115, StdDeviation = 10.14270]
> #[Max =522.36493, Total count= 317444]
> #[Buckets = 25, SubBuckets = 262144]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16039) FQL replay should have options to ignore DDL statements
[ https://issues.apache.org/jira/browse/CASSANDRA-16039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Capwell updated CASSANDRA-16039: -- Change Category: Operability Complexity: Low Hanging Fruit Fix Version/s: 4.0-rc Status: Open (was: Triage Needed) Marked as RC as DDL replay is unexpected, so can be confusing. > FQL replay should have options to ignore DDL statements > --- > > Key: CASSANDRA-16039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16039 > Project: Cassandra > Issue Type: Improvement > Components: Tool/fql >Reporter: David Capwell >Priority: Normal > Fix For: 4.0-rc > > > FQL logs will contain DDL statements made on the host, and the replay tool > will attempt to replay them; this can be unsafe in general so should have an > option to filter them out. > Thinking about it, since DDL replay isn’t obvious, we should most likely > default to false and have a flag to allow DDL replay as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-16039) FQL replay should have options to ignore DDL statements
David Capwell created CASSANDRA-16039: - Summary: FQL replay should have options to ignore DDL statements Key: CASSANDRA-16039 URL: https://issues.apache.org/jira/browse/CASSANDRA-16039 Project: Cassandra Issue Type: Improvement Components: Tool/fql Reporter: David Capwell FQL logs will contain DDL statements made on the host, and the replay tool will attempt to replay them; this can be unsafe in general so should have an option to filter them out. Thinking about it, since DDL replay isn’t obvious, we should most likely default to false and have a flag to allow DDL replay as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15393) Add byte array backed cells
[
https://issues.apache.org/jira/browse/CASSANDRA-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17172612#comment-17172612
]
Caleb Rackliffe edited comment on CASSANDRA-15393 at 8/7/20, 9:18 PM:
--
[~bdeggleston] I made a pass at a rebase, given the changes made in trunk
around CASSANDRA-15400, CASSANDRA-15461, and CASSANDRA-14365 since we last
touched this. You can find it
[here|https://github.com/apache/cassandra/pull/712] (and a link to a CircleCI
run is in the comments).
Notes:
- {{CompactionAllocationTest}} already exists on trunk, so I more or less took
its modifications during conflict resolution.
- I've
[preserved|https://github.com/apache/cassandra/pull/712/files#diff-e65bcd2d398327679a75a9da82473635R45]
the FIXME in the digest update.
- There are numerous places where I've added proper type/generics information
to method signatures, but I stopped, as that would have been a pretty big task
before we've even gotten complete agreement on the approach in general.
- I think I've implemented {{ClusteringPrefix#minimize()}} reasonably across
the native, array-backed, and bb-backed classes, but that's an area for us to
review.
With all of that cleaned up, I'll start on actually reviewing the patch :)
was (Author: maedhroz):
[~bdeggleston] I made a pass at a rebase, given the changes made in trunk
around CASSANDRA-15400, CASSANDRA-15461, and CASSANDRA-14365 since we last
touched this. You can find it
[here|https://github.com/apache/cassandra/pull/712] (and a link to a CircleCI
run is in the comments).
Notes:
- {{CompactionAllocationTest}} already exists on trunk, so I more or less took
its modifications during conflict resolution.
- I've
[preserved|https://github.com/apache/cassandra/pull/712/files#diff-e65bcd2d398327679a75a9da82473635R45]
the FIXME in the digest update.
- There are numerous places where I've added proper type/generics information
to method signatures. It probably inflated the diff a bit.
- I think I've implemented {{ClusteringPrefix#minimize()}} reasonably across
the native, array-backed, and bb-backed classes, but that's an area for us to
review.
With all of that cleaned up, I'll start on actually reviewing the patch :)
> Add byte array backed cells
> ---
>
> Key: CASSANDRA-15393
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15393
> Project: Cassandra
> Issue Type: Sub-task
> Components: Local/Compaction
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 4.0-beta
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We currently materialize all values as on heap byte buffers. Byte buffers
> have a fairly high overhead given how frequently they’re used, and on the
> compaction and local read path we don’t do anything that needs them. Use of
> byte buffer methods only happens on the coordinator. Using cells that are
> backed by byte arrays instead in these situations reduces compaction and read
> garbage up to 22% in many cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16036) Add flag to disable chunk cache and disable by default
[
https://issues.apache.org/jira/browse/CASSANDRA-16036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17173446#comment-17173446
]
David Capwell commented on CASSANDRA-16036:
---
[~jasonstack] was hoping I could get your feedback on this patch? From my
testing, if there are writes going on the cluster the cache hit ratio tanks and
chunk cache gets more costly than having it disabled; most workloads I see are
write heavy, hence the default I went with.
> Add flag to disable chunk cache and disable by default
> --
>
> Key: CASSANDRA-16036
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16036
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: Histogram-11.png
>
>
> Chunk cache is enabled by default and doesn’t have a flag to disable without
> impacting networking. In performance testing 4.0 against 3.0 I found that
> reads were slower in 4.0 and after profiling found that the ChunkCache was
> partially to blame; after disabling the chunk cache, read performance had
> improved.
> {code}
> 40_w_cc-selects.hdr
> #[Mean= 11.50063, StdDeviation = 13.44014]
> #[Max =482.41254, Total count= 316477]
> #[Buckets = 25, SubBuckets = 262144]
> 40_wo_cc-selects.hdr
> #[Mean= 9.82115, StdDeviation = 10.14270]
> #[Max =522.36493, Total count= 317444]
> #[Buckets = 25, SubBuckets = 262144]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16036) Add flag to disable chunk cache and disable by default
[
https://issues.apache.org/jira/browse/CASSANDRA-16036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Capwell updated CASSANDRA-16036:
--
Summary: Add flag to disable chunk cache and disable by default (was: Add
flag to disable chunk cache and disable by defaul)
> Add flag to disable chunk cache and disable by default
> --
>
> Key: CASSANDRA-16036
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16036
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: Histogram-11.png
>
>
> Chunk cache is enabled by default and doesn’t have a flag to disable without
> impacting networking. In performance testing 4.0 against 3.0 I found that
> reads were slower in 4.0 and after profiling found that the ChunkCache was
> partially to blame; after disabling the chunk cache, read performance had
> improved.
> {code}
> 40_w_cc-selects.hdr
> #[Mean= 11.50063, StdDeviation = 13.44014]
> #[Max =482.41254, Total count= 316477]
> #[Buckets = 25, SubBuckets = 262144]
> 40_wo_cc-selects.hdr
> #[Mean= 9.82115, StdDeviation = 10.14270]
> #[Max =522.36493, Total count= 317444]
> #[Buckets = 25, SubBuckets = 262144]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15971) full query log needs improvement
[
https://issues.apache.org/jira/browse/CASSANDRA-15971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17173429#comment-17173429
]
David Capwell commented on CASSANDRA-15971:
---
Spoke in slack, dumping here.
There was an assumption that replay should include schema changes, but since
Cassandra isn't good with schema changes (drop table, recreate same table) that
is unsafe; the more common use case for replay is to replay against a different
cluster. Speaking to Lorina, she will improve the example to show this.
> full query log needs improvement
>
>
> Key: CASSANDRA-15971
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15971
> Project: Cassandra
> Issue Type: Improvement
> Components: Documentation/Website, Tool/fql
>Reporter: Jonathan Shook
>Assignee: Lorina Poland
>Priority: Normal
> Labels: pull-request-available
> Attachments: st1.txt
>
>
> When trying out full query logging as a possible integration for nosqlbench
> usage, I ran across many issues which would make it painful for users. Since
> there were several, they will be added to a single issue for now. This issue
> can be broken up if needed.
>
> FQL doesn't work on my system, even though it says it is logging queries.
> With the following configuration in cassandra.yaml:
>
> {noformat}
> full_query_logging_options:
> log_dir: /REDACTED/fullquerylogs
> roll_cycle: HOURLY
> block: true
> max_queue_weight: 268435456 # 256 MiB
> max_log_size: 17179869184 # 16 GiB
> ## archive command is "/path/to/script.sh %path" where %path is replaced
> with the file being rolled:
> # archive_command:
> # max_archive_retries: 10
> {noformat}
> which appears to be the minimal configuration needed to enable fql, only a
> single file `directory-listing.cq4t` is created, which is a 64K sized file of
> zeroes.
>
>
> Calling bin/nodetool enablefullquerylog throws an error initially.
> [jshook@cass4 bin]$ ./nodetool enablefullquerylog
>
> {noformat}
> error: sun.nio.ch.FileChannelImpl.map0(int,long,long)
> -- StackTrace --
> java.lang.NoSuchMethodException:
> sun.nio.ch.FileChannelImpl.map0(int,long,long)
> at java.base/java.lang.Class.getDeclaredMethod(Class.java:2553)
> at net.openhft.chronicle.core.OS.lambda$static$0(OS.java:51)
> at
> net.openhft.chronicle.core.ClassLocal.computeValue(ClassLocal.java:53){noformat}
> (full stack trace attached to this ticket)
>
> Subsequent calls produce normal output:
>
> {noformat}
> [jshook@cass4 c4b1]$ bin/nodetool enablefullquerylog
> nodetool: Already logging to /home/jshook/c4b1/data/fullquerylogs
> See 'nodetool help' or 'nodetool help '.{noformat}
>
>
> nodetool missing getfullquerylog makes it difficult to verify current
> fullquerylog state without changing it. The conventions for nodetool commands
> should be followed to avoid confusing users.
>
> (maybe)
> {noformat}
> tools/bin/fqltool help{noformat}
> should print out help for all fqltool commands rather than simply repeating
> the default The most commonly used fqltool commands are..
>
> [https://cassandra.apache.org/doc/latest/new/fqllogging.html] is
> malformatted, mixing the appearance of configuration with comments, which is
> confusing at best.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15971) full query log needs improvement
[
https://issues.apache.org/jira/browse/CASSANDRA-15971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Capwell updated CASSANDRA-15971:
--
Reviewers: David Capwell, Ekaterina Dimitrova (was: Ekaterina Dimitrova)
> full query log needs improvement
>
>
> Key: CASSANDRA-15971
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15971
> Project: Cassandra
> Issue Type: Improvement
> Components: Documentation/Website, Tool/fql
>Reporter: Jonathan Shook
>Assignee: Lorina Poland
>Priority: Normal
> Labels: pull-request-available
> Attachments: st1.txt
>
>
> When trying out full query logging as a possible integration for nosqlbench
> usage, I ran across many issues which would make it painful for users. Since
> there were several, they will be added to a single issue for now. This issue
> can be broken up if needed.
>
> FQL doesn't work on my system, even though it says it is logging queries.
> With the following configuration in cassandra.yaml:
>
> {noformat}
> full_query_logging_options:
> log_dir: /REDACTED/fullquerylogs
> roll_cycle: HOURLY
> block: true
> max_queue_weight: 268435456 # 256 MiB
> max_log_size: 17179869184 # 16 GiB
> ## archive command is "/path/to/script.sh %path" where %path is replaced
> with the file being rolled:
> # archive_command:
> # max_archive_retries: 10
> {noformat}
> which appears to be the minimal configuration needed to enable fql, only a
> single file `directory-listing.cq4t` is created, which is a 64K sized file of
> zeroes.
>
>
> Calling bin/nodetool enablefullquerylog throws an error initially.
> [jshook@cass4 bin]$ ./nodetool enablefullquerylog
>
> {noformat}
> error: sun.nio.ch.FileChannelImpl.map0(int,long,long)
> -- StackTrace --
> java.lang.NoSuchMethodException:
> sun.nio.ch.FileChannelImpl.map0(int,long,long)
> at java.base/java.lang.Class.getDeclaredMethod(Class.java:2553)
> at net.openhft.chronicle.core.OS.lambda$static$0(OS.java:51)
> at
> net.openhft.chronicle.core.ClassLocal.computeValue(ClassLocal.java:53){noformat}
> (full stack trace attached to this ticket)
>
> Subsequent calls produce normal output:
>
> {noformat}
> [jshook@cass4 c4b1]$ bin/nodetool enablefullquerylog
> nodetool: Already logging to /home/jshook/c4b1/data/fullquerylogs
> See 'nodetool help' or 'nodetool help '.{noformat}
>
>
> nodetool missing getfullquerylog makes it difficult to verify current
> fullquerylog state without changing it. The conventions for nodetool commands
> should be followed to avoid confusing users.
>
> (maybe)
> {noformat}
> tools/bin/fqltool help{noformat}
> should print out help for all fqltool commands rather than simply repeating
> the default The most commonly used fqltool commands are..
>
> [https://cassandra.apache.org/doc/latest/new/fqllogging.html] is
> malformatted, mixing the appearance of configuration with comments, which is
> confusing at best.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15971) full query log needs improvement
[
https://issues.apache.org/jira/browse/CASSANDRA-15971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17173409#comment-17173409
]
David Capwell commented on CASSANDRA-15971:
---
Left comments on the PR:
https://github.com/apache/cassandra/pull/705#pullrequestreview-463503631
Overall LGTM, only small things.
> full query log needs improvement
>
>
> Key: CASSANDRA-15971
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15971
> Project: Cassandra
> Issue Type: Improvement
> Components: Documentation/Website, Tool/fql
>Reporter: Jonathan Shook
>Assignee: Lorina Poland
>Priority: Normal
> Labels: pull-request-available
> Attachments: st1.txt
>
>
> When trying out full query logging as a possible integration for nosqlbench
> usage, I ran across many issues which would make it painful for users. Since
> there were several, they will be added to a single issue for now. This issue
> can be broken up if needed.
>
> FQL doesn't work on my system, even though it says it is logging queries.
> With the following configuration in cassandra.yaml:
>
> {noformat}
> full_query_logging_options:
> log_dir: /REDACTED/fullquerylogs
> roll_cycle: HOURLY
> block: true
> max_queue_weight: 268435456 # 256 MiB
> max_log_size: 17179869184 # 16 GiB
> ## archive command is "/path/to/script.sh %path" where %path is replaced
> with the file being rolled:
> # archive_command:
> # max_archive_retries: 10
> {noformat}
> which appears to be the minimal configuration needed to enable fql, only a
> single file `directory-listing.cq4t` is created, which is a 64K sized file of
> zeroes.
>
>
> Calling bin/nodetool enablefullquerylog throws an error initially.
> [jshook@cass4 bin]$ ./nodetool enablefullquerylog
>
> {noformat}
> error: sun.nio.ch.FileChannelImpl.map0(int,long,long)
> -- StackTrace --
> java.lang.NoSuchMethodException:
> sun.nio.ch.FileChannelImpl.map0(int,long,long)
> at java.base/java.lang.Class.getDeclaredMethod(Class.java:2553)
> at net.openhft.chronicle.core.OS.lambda$static$0(OS.java:51)
> at
> net.openhft.chronicle.core.ClassLocal.computeValue(ClassLocal.java:53){noformat}
> (full stack trace attached to this ticket)
>
> Subsequent calls produce normal output:
>
> {noformat}
> [jshook@cass4 c4b1]$ bin/nodetool enablefullquerylog
> nodetool: Already logging to /home/jshook/c4b1/data/fullquerylogs
> See 'nodetool help' or 'nodetool help '.{noformat}
>
>
> nodetool missing getfullquerylog makes it difficult to verify current
> fullquerylog state without changing it. The conventions for nodetool commands
> should be followed to avoid confusing users.
>
> (maybe)
> {noformat}
> tools/bin/fqltool help{noformat}
> should print out help for all fqltool commands rather than simply repeating
> the default The most commonly used fqltool commands are..
>
> [https://cassandra.apache.org/doc/latest/new/fqllogging.html] is
> malformatted, mixing the appearance of configuration with comments, which is
> confusing at best.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15971) full query log needs improvement
[
https://issues.apache.org/jira/browse/CASSANDRA-15971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17173384#comment-17173384
]
David Capwell commented on CASSANDRA-15971:
---
sorry for the delay, looking now.
> full query log needs improvement
>
>
> Key: CASSANDRA-15971
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15971
> Project: Cassandra
> Issue Type: Improvement
> Components: Documentation/Website, Tool/fql
>Reporter: Jonathan Shook
>Assignee: Lorina Poland
>Priority: Normal
> Labels: pull-request-available
> Attachments: st1.txt
>
>
> When trying out full query logging as a possible integration for nosqlbench
> usage, I ran across many issues which would make it painful for users. Since
> there were several, they will be added to a single issue for now. This issue
> can be broken up if needed.
>
> FQL doesn't work on my system, even though it says it is logging queries.
> With the following configuration in cassandra.yaml:
>
> {noformat}
> full_query_logging_options:
> log_dir: /REDACTED/fullquerylogs
> roll_cycle: HOURLY
> block: true
> max_queue_weight: 268435456 # 256 MiB
> max_log_size: 17179869184 # 16 GiB
> ## archive command is "/path/to/script.sh %path" where %path is replaced
> with the file being rolled:
> # archive_command:
> # max_archive_retries: 10
> {noformat}
> which appears to be the minimal configuration needed to enable fql, only a
> single file `directory-listing.cq4t` is created, which is a 64K sized file of
> zeroes.
>
>
> Calling bin/nodetool enablefullquerylog throws an error initially.
> [jshook@cass4 bin]$ ./nodetool enablefullquerylog
>
> {noformat}
> error: sun.nio.ch.FileChannelImpl.map0(int,long,long)
> -- StackTrace --
> java.lang.NoSuchMethodException:
> sun.nio.ch.FileChannelImpl.map0(int,long,long)
> at java.base/java.lang.Class.getDeclaredMethod(Class.java:2553)
> at net.openhft.chronicle.core.OS.lambda$static$0(OS.java:51)
> at
> net.openhft.chronicle.core.ClassLocal.computeValue(ClassLocal.java:53){noformat}
> (full stack trace attached to this ticket)
>
> Subsequent calls produce normal output:
>
> {noformat}
> [jshook@cass4 c4b1]$ bin/nodetool enablefullquerylog
> nodetool: Already logging to /home/jshook/c4b1/data/fullquerylogs
> See 'nodetool help' or 'nodetool help '.{noformat}
>
>
> nodetool missing getfullquerylog makes it difficult to verify current
> fullquerylog state without changing it. The conventions for nodetool commands
> should be followed to avoid confusing users.
>
> (maybe)
> {noformat}
> tools/bin/fqltool help{noformat}
> should print out help for all fqltool commands rather than simply repeating
> the default The most commonly used fqltool commands are..
>
> [https://cassandra.apache.org/doc/latest/new/fqllogging.html] is
> malformatted, mixing the appearance of configuration with comments, which is
> confusing at best.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16036) Add flag to disable chunk cache and disable by defaul
[
https://issues.apache.org/jira/browse/CASSANDRA-16036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17172892#comment-17172892
]
David Capwell edited comment on CASSANDRA-16036 at 8/7/20, 4:49 PM:
The test performed was 11% read of slices on a clustering key, so 89% write;
workload from prod.
What I saw in CPU profiles was that the chunk cache was a decent part of
compaction and the read path, but not in a positive way. With compaction also
populating the cache, it caused a constant churn from the cache, but all reads
had to pay the cost for it.
was (Author: dcapwell):
The test performed was 11% read of slices on a clustering key, so 89% write;
workload from prod.
What I was in the CPU profiles was that the chunk cache was a good chunk of
compaction and the read path, but not in a positive way. With compaction also
populating the cache, it caused a constant churn from the cache, but all reads
had to pay the cost for it.
> Add flag to disable chunk cache and disable by defaul
> -
>
> Key: CASSANDRA-16036
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16036
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: Histogram-11.png
>
>
> Chunk cache is enabled by default and doesn’t have a flag to disable without
> impacting networking. In performance testing 4.0 against 3.0 I found that
> reads were slower in 4.0 and after profiling found that the ChunkCache was
> partially to blame; after disabling the chunk cache, read performance had
> improved.
> {code}
> 40_w_cc-selects.hdr
> #[Mean= 11.50063, StdDeviation = 13.44014]
> #[Max =482.41254, Total count= 316477]
> #[Buckets = 25, SubBuckets = 262144]
> 40_wo_cc-selects.hdr
> #[Mean= 9.82115, StdDeviation = 10.14270]
> #[Max =522.36493, Total count= 317444]
> #[Buckets = 25, SubBuckets = 262144]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16036) Add flag to disable chunk cache and disable by defaul
[
https://issues.apache.org/jira/browse/CASSANDRA-16036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17173292#comment-17173292
]
David Capwell commented on CASSANDRA-16036:
---
CI is Yellow; known failing tests only.
> Add flag to disable chunk cache and disable by defaul
> -
>
> Key: CASSANDRA-16036
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16036
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: Histogram-11.png
>
>
> Chunk cache is enabled by default and doesn’t have a flag to disable without
> impacting networking. In performance testing 4.0 against 3.0 I found that
> reads were slower in 4.0 and after profiling found that the ChunkCache was
> partially to blame; after disabling the chunk cache, read performance had
> improved.
> {code}
> 40_w_cc-selects.hdr
> #[Mean= 11.50063, StdDeviation = 13.44014]
> #[Max =482.41254, Total count= 316477]
> #[Buckets = 25, SubBuckets = 262144]
> 40_wo_cc-selects.hdr
> #[Mean= 9.82115, StdDeviation = 10.14270]
> #[Max =522.36493, Total count= 317444]
> #[Buckets = 25, SubBuckets = 262144]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16036) Add flag to disable chunk cache and disable by defaul
[
https://issues.apache.org/jira/browse/CASSANDRA-16036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17173291#comment-17173291
]
David Capwell commented on CASSANDRA-16036:
---
Rerunning the test with chunk cache enabled, seeing the following for
hits/misses
{code}
OneMinuteRate
Hits
392.07075331154306
Misses
10004.73634589861
{code}
So a 3.8% cache hit rate, this makes sense to me since compaction keeps
populating the cache with files that are about to be deleted
> Add flag to disable chunk cache and disable by defaul
> -
>
> Key: CASSANDRA-16036
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16036
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: Histogram-11.png
>
>
> Chunk cache is enabled by default and doesn’t have a flag to disable without
> impacting networking. In performance testing 4.0 against 3.0 I found that
> reads were slower in 4.0 and after profiling found that the ChunkCache was
> partially to blame; after disabling the chunk cache, read performance had
> improved.
> {code}
> 40_w_cc-selects.hdr
> #[Mean= 11.50063, StdDeviation = 13.44014]
> #[Max =482.41254, Total count= 316477]
> #[Buckets = 25, SubBuckets = 262144]
> 40_wo_cc-selects.hdr
> #[Mean= 9.82115, StdDeviation = 10.14270]
> #[Max =522.36493, Total count= 317444]
> #[Buckets = 25, SubBuckets = 262144]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-16038) Add a getter for InstanceConfig parameters - in-jvm-dtests-api
[ https://issues.apache.org/jira/browse/CASSANDRA-16038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ekaterina Dimitrova reassigned CASSANDRA-16038: --- Assignee: (was: Ekaterina Dimitrova) > Add a getter for InstanceConfig parameters - in-jvm-dtests-api > -- > > Key: CASSANDRA-16038 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16038 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Ekaterina Dimitrova >Priority: Low > > In order to change the way config will be loaded (for reference > CASSANDRA-15234 ) a getter for the InstanceConfig parameters is needed > CC [~maedhroz] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16031) Run in-jvm upgrade dtests in ci-cassandra
[ https://issues.apache.org/jira/browse/CASSANDRA-16031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17173225#comment-17173225 ] David Capwell commented on CASSANDRA-16031: --- Didn't test, but validated the names matched; this LGTM +1 > Run in-jvm upgrade dtests in ci-cassandra > - > > Key: CASSANDRA-16031 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16031 > Project: Cassandra > Issue Type: Improvement > Components: CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.0-rc > > Time Spent: 40m > Remaining Estimate: 0h > > Add jvm-dtest-upgrade to ci-cassandra.a.o -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16029) Create better Apache Cassandra 4.0 docs
[ https://issues.apache.org/jira/browse/CASSANDRA-16029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lorina Poland updated CASSANDRA-16029: -- Change Category: Performance Complexity: Challenging Fix Version/s: 4.0 Reviewers: Jon Haddad, Mick Semb Wever Status: Open (was: Triage Needed) > Create better Apache Cassandra 4.0 docs > --- > > Key: CASSANDRA-16029 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16029 > Project: Cassandra > Issue Type: Improvement > Components: Documentation/Website >Reporter: Lorina Poland >Assignee: Lorina Poland >Priority: Normal > Fix For: 4.0 > > Attachments: CurrentDocs.png, NewImprovedDocs.png > > > The proof of concept to create new Documentation showed that a combination of > the static site generator (SSG) Antora + Asciidoc files would provide the > most flexibility in redesigning the Apache C* OSS pages. > Current proof of concept: [https://polandll.github.io/site/Cassandra/4.0/] > To update the docs, the tools to write needed an update, too, to provide a > better UX and modern look and feel. The old tools (reStructured Text, Sphinx) > will be replaced with the new tools ([AsciiDoc|http://asciidoc.org], > [Antora|http://antora.org]). > See the attachments for screenshots of the current docs and the POC docs. > The advantages of asciidoc+antora: > * Rich markdown language that is directly editable. > * Good organizational structure for docs files > * Flexible build/publishing capabilities, including content from multiple > repositories and branches > * Flexible web UI, built separately from the doc files and static site. > * JS extensions that enable additional functionality. > To complete the conversion, the following steps are required: > * Pandoc used for conversion of rSt files to adoc files > ** Followed by significant manual editing to fix the errors left over after > conversion > * Creation of new antora files (site.yml, antora.yml, CSS and layout) > ** Integration with plug-ins (lunr for search, tabs for additional codeblock > capabilities) > ** Finish the UI to match current Apache C* UI, or engage designer to do a > new design > *** Put the ASF pulldown in the main header banner, like Apache Spark does > ([https://spark.apache.org/downloads.html]) > * Modification of build scripts to work with Antora > ** Modification of python scripts that generation YAML and nodetool docs > *** Make sure that these scripts are run for each version > ** Modification of cassandra/doc Makefile > ** Dockerfile and docker-entrypoint.sh files to use Docker for documentation > build > * Modification of cassandra-website to work with Antora > ** Dockerfile and docker-entrypoint.sh files to use Docker for documentation > build > * Figure out how to handle older versions that are not converted to asciidoc > now > ** Put in TBD? Copy 4.0 branch with note to rewrite later? > * Figure out why tabs-block antora extension is working locally but not in > GH pages (may or may not be important) > Other tasks: > * Complete conversion, plus editing the current docs before Apache C* 4.0 GA > * Further improvements for docs organization > ** > [https://docs.google.com/document/d/1aeNtgyPAsKcNa0GSKvl2ywlFEj30714ry4Sb5turWeE/edit?usp=sharing] > ** Get some sections out of Docs, like Developing Code info -> Community > * Solve the issue of CQL highlighting/lexer/parser issue > ** Need CQL to be submitted to pygments > * Addition of more useful documentation - tutorials, how-to guides, separate > reference guide > Current work: > [https://github.com/polandll/cassandra/blob/doc_redo_asciidoc/|https://github.com/polandll/cassandra/blob/doc_redo_asciidoc/doc/Dockerfile] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-16038) Add a getter for InstanceConfig parameters - in-jvm-dtests-api
Ekaterina Dimitrova created CASSANDRA-16038: --- Summary: Add a getter for InstanceConfig parameters - in-jvm-dtests-api Key: CASSANDRA-16038 URL: https://issues.apache.org/jira/browse/CASSANDRA-16038 Project: Cassandra Issue Type: Task Reporter: Ekaterina Dimitrova Assignee: Ekaterina Dimitrova In order to change the way config will be loaded (for reference CASSANDRA-15234 ) a getter for the InstanceConfig parameters is needed CC [~maedhroz] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16038) Add a getter for InstanceConfig parameters - in-jvm-dtests-api
[ https://issues.apache.org/jira/browse/CASSANDRA-16038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ekaterina Dimitrova updated CASSANDRA-16038: Change Category: Quality Assurance Complexity: Low Hanging Fruit Component/s: Test/dtest Priority: Low (was: Normal) Status: Open (was: Triage Needed) > Add a getter for InstanceConfig parameters - in-jvm-dtests-api > -- > > Key: CASSANDRA-16038 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16038 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Ekaterina Dimitrova >Assignee: Ekaterina Dimitrova >Priority: Low > > In order to change the way config will be loaded (for reference > CASSANDRA-15234 ) a getter for the InstanceConfig parameters is needed > CC [~maedhroz] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16037) Add a getter for InstanceConfig parameters - in-jvm-dtest-api
[ https://issues.apache.org/jira/browse/CASSANDRA-16037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ekaterina Dimitrova updated CASSANDRA-16037: Resolution: Not A Bug Status: Resolved (was: Triage Needed) > Add a getter for InstanceConfig parameters - in-jvm-dtest-api > - > > Key: CASSANDRA-16037 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16037 > Project: Cassandra > Issue Type: Bug >Reporter: Ekaterina Dimitrova >Priority: Normal > > In order to change the way config will be loaded (for reference > CASSANDRA-15234 ) a getter for the InstanceConfig parameters is needed -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-16037) Add a getter for InstanceConfig parameters - in-jvm-dtest-api
Ekaterina Dimitrova created CASSANDRA-16037: --- Summary: Add a getter for InstanceConfig parameters - in-jvm-dtest-api Key: CASSANDRA-16037 URL: https://issues.apache.org/jira/browse/CASSANDRA-16037 Project: Cassandra Issue Type: Bug Reporter: Ekaterina Dimitrova In order to change the way config will be loaded (for reference CASSANDRA-15234 ) a getter for the InstanceConfig parameters is needed -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15986) Repair tests flakiness
[ https://issues.apache.org/jira/browse/CASSANDRA-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17173152#comment-17173152 ] Ekaterina Dimitrova edited comment on CASSANDRA-15986 at 8/7/20, 1:53 PM: -- Sure! I just created a VM in VirtualBox with Ubuntu guest OS - [ubuntu-18.04.4-desktop-amd64.iso |https://releases.ubuntu.com/18.04/]. I set RAM to 4096MB and disk space to 25GB instead of 10GB, everything else is just default config. was (Author: e.dimitrova): Sure! I just created a VM in VirtualBox with Ubuntu guest OS - [ubuntu-18.04.4-desktop-amd64.iso |https://releases.ubuntu.com/18.04/]. I set RAM to 4096MB and disk space to 25GB instead of 10GB, everything else is just default. > Repair tests flakiness > -- > > Key: CASSANDRA-15986 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15986 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Berenguer Blasi >Priority: Normal > Fix For: 4.0-beta > > > Repair tests come up in test failure reports every now and then. I have tried > to repro the > [latest|https://ci-cassandra.apache.org/job/Cassandra-trunk/241/testReport/junit/dtest-novnode.repair_tests.repair_test/TestRepair/test_simple_sequential_repair/] > locally 100 times with no luck. > Still from experience from fixing other flaky tests I have some intuition > where the problems may lie. The proposed fix should add no harm if merged. We > can reopen the ticket if repair tests keep failing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15986) Repair tests flakiness
[ https://issues.apache.org/jira/browse/CASSANDRA-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17173152#comment-17173152 ] Ekaterina Dimitrova commented on CASSANDRA-15986: - Sure! I just created a VM in VirtualBox with Ubuntu guest OS - [ubuntu-18.04.4-desktop-amd64.iso |https://releases.ubuntu.com/18.04/]. I set RAM to 4096MB and disk space to 25GB instead of 10GB, everything else is just default. > Repair tests flakiness > -- > > Key: CASSANDRA-15986 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15986 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Berenguer Blasi >Priority: Normal > Fix For: 4.0-beta > > > Repair tests come up in test failure reports every now and then. I have tried > to repro the > [latest|https://ci-cassandra.apache.org/job/Cassandra-trunk/241/testReport/junit/dtest-novnode.repair_tests.repair_test/TestRepair/test_simple_sequential_repair/] > locally 100 times with no luck. > Still from experience from fixing other flaky tests I have some intuition > where the problems may lie. The proposed fix should add no harm if merged. We > can reopen the ticket if repair tests keep failing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15990) Running CQL command with non-ASCII values raises UnicodeDecodeError
[
https://issues.apache.org/jira/browse/CASSANDRA-15990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ekaterina Dimitrova updated CASSANDRA-15990:
Complexity: Low Hanging Fruit (was: Normal)
> Running CQL command with non-ASCII values raises UnicodeDecodeError
> ---
>
> Key: CASSANDRA-15990
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15990
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/cqlsh
>Reporter: Joseph Chu
>Assignee: Ekaterina Dimitrova
>Priority: Normal
> Fix For: 4.0-beta
>
>
> There are INSERT statements that contains non-ASCII values that have run fine
> in Cassandra 3.11, but now raises a UnicodeDecodeError when I try executing
> them in 4.0-alpha4 and 4.0-beta1.
> Example input and output:
> {code:java}
> echo $LANG
> en_US.UTF-8
> $ cqlsh --debug
> Using CQL driver: '/usr/share/cassandra/bin/../lib/cassandra-driver-internal-only-3.23.0.post0-1a184b99.zip/cassandra-driver-3.23.0.post0-1a184b99/cassandra/__init__.py'>
> Using connect timeout: 5 seconds
> Using 'utf-8' encoding
> Using ssl: False
> Connected to Cassandra Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 4.0-beta1 | CQL spec 3.4.5 | Native protocol v4]
> Use HELP for help.
> cqlsh> CREATE KEYSPACE killr_video WITH replication = {'class':
> 'NetworkTopologyStrategy', 'DC-Houston': 1};
> cqlsh> USE killr_video;
> cqlsh:killr_video> CREATE TABLE movies_by_genre ( genre TEXT, title TEXT,
> year INT, duration INT, avg_rating FLOAT, country TEXT, PRIMARY KEY ((genre),
> title, year));
> cqlsh:killr_video> INSERT INTO movies_by_genre (genre, title, year, duration,
> avg_rating, country)
> ... VALUES ('Action', 'The Extraordinary Adventures of Adèle Blanc-Sec',
> 2010, 107, 6.30, 'France');
> Traceback (most recent call last):
> File "/usr/share/cassandra/bin/cqlsh.py", line 937, in onecmd
> self.handle_statement(st, statementtext)
> File "/usr/share/cassandra/bin/cqlsh.py", line 962, in handle_statement
> readline.add_history(new_hist)
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe8' in position
> 134: ordinal not in range(128){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15991) 15583 - Add UX tests to intree LHF tooling
[ https://issues.apache.org/jira/browse/CASSANDRA-15991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17173047#comment-17173047 ] Berenguer Blasi edited comment on CASSANDRA-15991 at 8/7/20, 10:21 AM: --- [~samt]I have put this one for review finally. This is just cmd line params happy path unit testing to seed the individual test tickets per tool created in CASSANDRA-15583. Is is important though as it sets up the foundation to build upon. I already have 3 of those tool specific tickets almost ready to go into review once this is merged. If you're busy or going to be on holidays let me know and I'll look for another reviewer is you prefer? Thx in advance. Edit: CASSANDRA-15956 is minimal and I folded it in this PR as I needed it as I was building upon it. was (Author: bereng): [~samt]I have put this one for review finally. This is just cmd line params happy path unit testing to seed the individual test tickets per tool created in CASSANDRA-15583. Is is important though as it sets up the foundation to build upon. I already have 3 of those tool specific tickets almost ready to go into review once this is merged. If you're busy or going to be on holidays let me know and I'll look for another reviewer is you prefer? Thx in advance. > 15583 - Add UX tests to intree LHF tooling > -- > > Key: CASSANDRA-15991 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15991 > Project: Cassandra > Issue Type: Improvement > Components: Test/unit >Reporter: Berenguer Blasi >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 4.0-beta > > > As per CASSANDRA-15583 many in tree tools lack proper UX tooling: mandatory > params are indeed mandatory, 'help' produces an actual help, return codes etc > This ticket is an attempt to add it to those tools that classify as LHF. > Other tools such as nodetool, with many sub-commands, deserve a separate > ticket of their own -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15991) 15583 - Add UX tests to intree LHF tooling
[ https://issues.apache.org/jira/browse/CASSANDRA-15991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17173047#comment-17173047 ] Berenguer Blasi commented on CASSANDRA-15991: - [~samt]I have put this one for review finally. This is just cmd line params happy path unit testing to seed the individual test tickets per tool created in CASSANDRA-15583. Is is important though as it sets up the foundation to build upon. I already have 3 of those tool specific tickets almost ready to go into review once this is merged. If you're busy or going to be on holidays let me know and I'll look for another reviewer is you prefer? Thx in advance. > 15583 - Add UX tests to intree LHF tooling > -- > > Key: CASSANDRA-15991 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15991 > Project: Cassandra > Issue Type: Improvement > Components: Test/unit >Reporter: Berenguer Blasi >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 4.0-beta > > > As per CASSANDRA-15583 many in tree tools lack proper UX tooling: mandatory > params are indeed mandatory, 'help' produces an actual help, return codes etc > This ticket is an attempt to add it to those tools that classify as LHF. > Other tools such as nodetool, with many sub-commands, deserve a separate > ticket of their own -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15991) 15583 - Add UX tests to intree LHF tooling
[ https://issues.apache.org/jira/browse/CASSANDRA-15991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Berenguer Blasi updated CASSANDRA-15991: Test and Documentation Plan: See PR Status: Patch Available (was: In Progress) > 15583 - Add UX tests to intree LHF tooling > -- > > Key: CASSANDRA-15991 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15991 > Project: Cassandra > Issue Type: Improvement > Components: Test/unit >Reporter: Berenguer Blasi >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 4.0-beta > > > As per CASSANDRA-15583 many in tree tools lack proper UX tooling: mandatory > params are indeed mandatory, 'help' produces an actual help, return codes etc > This ticket is an attempt to add it to those tools that classify as LHF. > Other tools such as nodetool, with many sub-commands, deserve a separate > ticket of their own -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15393) Add byte array backed cells
[
https://issues.apache.org/jira/browse/CASSANDRA-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17173037#comment-17173037
]
Benedict Elliott Smith edited comment on CASSANDRA-15393 at 8/7/20, 10:03 AM:
--
{quote}But introducing an API that's error prone is dangerous.
{quote}
This particular issue can be somewhat resolved by using generic parameters for
methods that accept the object and accessor, including the constructor of any
object holding them, e.g.
{code:java}
public interface Accessor {}
public class SomethingWithData
{
final Object data;
final Accessor accessor;
public SomethingWithData(D data, Accessor accessor) {
this.data = data; this.accessor = accessor; }
} {code}
This avoids polluting the codebase with type parameters, but ensures the first
supplier of the Object+Accessor is safe, and later supplies are safe if they
only propagate pairs produced in this way.
We could also probably generify the objects themselves, I don't think it would
be too painful - but this at least is an easy and less invasive improvement.
{quote}The underlying issue IMO is not {{ByteBuffer}} itself but the vast
amount of those intermediate and often tiny BB instances.
{quote}
I don't have a strong opinion on the overall structural impact of this change;
I haven't thought through the various options. However I don't think we
produce all that many intermediate {{ByteBuffer}} specifically, and though we
do produce a lot of intermediate objects in many cases, there are workloads
where this isn't the issue, and object creation is still a problem.
When it comes to compaction, my preferred approach would be to avoid
materialising on heap at all - but the change here helps both compaction and
reads/writes, so it doesn't have to be either/or, and we should definitely be
trying to reduce our heap footprint in general.
was (Author: benedict):
{quote}But introducing an API that's error prone is dangerous. But introducing
an API that's error prone is dangerous.
{quote}
This particular issue can be somewhat resolved by using generic parameters for
methods that accept the object and accessor, including the constructor of any
object holding them, e.g.
{code:java}
public interface Accessor {}
public class SomethingWithData
{
final Object data;
final Accessor accessor;
public SomethingWithData(D data, Accessor accessor) {
this.data = data; this.accessor = accessor; }
} {code}
This avoids polluting the codebase with type parameters, but ensures the first
supplier of the Object+Accessor is safe, and later supplies are safe if they
only propagate pairs produced in this way.
We could also probably generify the objects themselves, I don't think it would
be too painful - but this at least is an easy and less invasive improvement.
{quote}The underlying issue IMO is not {{ByteBuffer}} itself but the vast
amount of those intermediate and often tiny BB instances.
{quote}
I don't have a strong opinion on the overall structural impact of this change;
I haven't thought through the various options. However I don't think we
produce all that many intermediate {{ByteBuffer}} specifically, and though we
do produce a lot of intermediate objects in many cases, there are workloads
where this isn't the issue, and object creation is still a problem.
When it comes to compaction, my preferred approach would be to avoid
materialising on heap at all - but the change here helps both compaction and
reads/writes, so it doesn't have to be either/or, and we should definitely be
trying to reduce our heap footprint in general.
> Add byte array backed cells
> ---
>
> Key: CASSANDRA-15393
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15393
> Project: Cassandra
> Issue Type: Sub-task
> Components: Local/Compaction
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 4.0-beta
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We currently materialize all values as on heap byte buffers. Byte buffers
> have a fairly high overhead given how frequently they’re used, and on the
> compaction and local read path we don’t do anything that needs them. Use of
> byte buffer methods only happens on the coordinator. Using cells that are
> backed by byte arrays instead in these situations reduces compaction and read
> garbage up to 22% in many cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15393) Add byte array backed cells
[
https://issues.apache.org/jira/browse/CASSANDRA-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17173037#comment-17173037
]
Benedict Elliott Smith commented on CASSANDRA-15393:
{quote}But introducing an API that's error prone is dangerous. But introducing
an API that's error prone is dangerous.
{quote}
This particular issue can be somewhat resolved by using generic parameters for
methods that accept the object and accessor, including the constructor of any
object holding them, e.g.
{code:java}
public interface Accessor {}
public class SomethingWithData
{
final Object data;
final Accessor accessor;
public SomethingWithData(D data, Accessor accessor) {
this.data = data; this.accessor = accessor; }
} {code}
This avoids polluting the codebase with type parameters, but ensures the first
supplier of the Object+Accessor is safe, and later supplies are safe if they
only propagate pairs produced in this way.
We could also probably generify the objects themselves, I don't think it would
be too painful - but this at least is an easy and less invasive improvement.
{quote}The underlying issue IMO is not {{ByteBuffer}} itself but the vast
amount of those intermediate and often tiny BB instances.
{quote}
I don't have a strong opinion on the overall structural impact of this change;
I haven't thought through the various options. However I don't think we
produce all that many intermediate {{ByteBuffer}} specifically, and though we
do produce a lot of intermediate objects in many cases, there are workloads
where this isn't the issue, and object creation is still a problem.
When it comes to compaction, my preferred approach would be to avoid
materialising on heap at all - but the change here helps both compaction and
reads/writes, so it doesn't have to be either/or, and we should definitely be
trying to reduce our heap footprint in general.
> Add byte array backed cells
> ---
>
> Key: CASSANDRA-15393
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15393
> Project: Cassandra
> Issue Type: Sub-task
> Components: Local/Compaction
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 4.0-beta
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We currently materialize all values as on heap byte buffers. Byte buffers
> have a fairly high overhead given how frequently they’re used, and on the
> compaction and local read path we don’t do anything that needs them. Use of
> byte buffer methods only happens on the coordinator. Using cells that are
> backed by byte arrays instead in these situations reduces compaction and read
> garbage up to 22% in many cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16031) Run in-jvm upgrade dtests in ci-cassandra
[ https://issues.apache.org/jira/browse/CASSANDRA-16031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-16031: --- Test and Documentation Plan: ci-cassandra.a.o jobs Status: Patch Available (was: In Progress) Patches for [3.0|https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/cassandra-3.0_16031], [3.11|https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/cassandra-3.11_16031], and [trunk|https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/trunk_16031]. The individual successful builds for these jobs are at - [Cassandra-3.0-jvm-dtest-upgrade|https://ci-cassandra.apache.org/view/all/job/Cassandra-3.0-jvm-dtest-upgrade/] - [Cassandra-3.11-jvm-dtest-upgrade|https://ci-cassandra.apache.org/view/all/job/Cassandra-3.11-jvm-dtest-upgrade/] - [Cassandra-trunk-jvm-dtest-upgrade|https://ci-cassandra.apache.org/view/all/job/Cassandra-trunk-jvm-dtest-upgrade/] > Run in-jvm upgrade dtests in ci-cassandra > - > > Key: CASSANDRA-16031 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16031 > Project: Cassandra > Issue Type: Improvement > Components: CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.0-rc > > Time Spent: 40m > Remaining Estimate: 0h > > Add jvm-dtest-upgrade to ci-cassandra.a.o -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16031) Run in-jvm upgrade dtests in ci-cassandra
[ https://issues.apache.org/jira/browse/CASSANDRA-16031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-16031: --- Fix Version/s: (was: 2.2.x) > Run in-jvm upgrade dtests in ci-cassandra > - > > Key: CASSANDRA-16031 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16031 > Project: Cassandra > Issue Type: Improvement > Components: CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.0-rc > > Time Spent: 40m > Remaining Estimate: 0h > > Add jvm-dtest-upgrade to ci-cassandra.a.o -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15393) Add byte array backed cells
[
https://issues.apache.org/jira/browse/CASSANDRA-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17173009#comment-17173009
]
Robert Stupp commented on CASSANDRA-15393:
--
Thanks for tackling this! Heap pressure induced by {{ByteBuffer}} is definitely
a pita. I'm not sure though whether this patch is suitable to be included in
4.0 this late (beta stage) as it looks quite intrusive and is pretty big.
Haven't thoroughly reviewed it, but here are a few notes:
* The new *Accessor classes have constants for boolean (true/false)
representations, which are modifiable. IIRC there's been an issue in the past
when that (or a similar) constant was accidentally changed.
* The split of {{ByteBuffer}} into a pair of {{accessor + container}} is error
prone - e.g. accidentally passing a {{byte[]}} with the {{ByteBufferAccessor}}
or vice versa leads to "interesting" results.
The underlying issue IMO is not {{ByteBuffer}} itself but the vast amount of
those intermediate and often tiny BB instances. But introducing an API that's
error prone is dangerous.
It feels both more efficient and much safer to have an API & implementation
that actually _prevents_ (the vast majority of) these new objects. I.e. instead
of creating a new {{ByteBuffer}} (or "just" a new {{byte[]}} as in the PR) to
write some value (e.g. an {{int}}) to a target buffer, write that value
_directly_ into the target buffer. _How_ that's implemented is a different
topic, but I'd prefer to consider all the new developments in Java and make it
safe & transparent to use.
> Add byte array backed cells
> ---
>
> Key: CASSANDRA-15393
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15393
> Project: Cassandra
> Issue Type: Sub-task
> Components: Local/Compaction
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 4.0-beta
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We currently materialize all values as on heap byte buffers. Byte buffers
> have a fairly high overhead given how frequently they’re used, and on the
> compaction and local read path we don’t do anything that needs them. Use of
> byte buffer methods only happens on the coordinator. Using cells that are
> backed by byte arrays instead in these situations reduces compaction and read
> garbage up to 22% in many cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15986) Repair tests flakiness
[ https://issues.apache.org/jira/browse/CASSANDRA-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17172955#comment-17172955 ] Berenguer Blasi commented on CASSANDRA-15986: - I'd be interested to know how to build that slow Ubuntu VM (or can I download it somwhere/somehow?) so I don't have to reinvent the wheel trying which config is slow enough... > Repair tests flakiness > -- > > Key: CASSANDRA-15986 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15986 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Berenguer Blasi >Priority: Normal > Fix For: 4.0-beta > > > Repair tests come up in test failure reports every now and then. I have tried > to repro the > [latest|https://ci-cassandra.apache.org/job/Cassandra-trunk/241/testReport/junit/dtest-novnode.repair_tests.repair_test/TestRepair/test_simple_sequential_repair/] > locally 100 times with no luck. > Still from experience from fixing other flaky tests I have some intuition > where the problems may lie. The proposed fix should add no harm if merged. We > can reopen the ticket if repair tests keep failing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
