[jira] [Commented] (CASSANDRA-15446) Per-thread stack size is too small on aarch64 CentOS
[ https://issues.apache.org/jira/browse/CASSANDRA-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266524#comment-17266524 ] zhaorenhai commented on CASSANDRA-15446: What do you think about changing Xss256k to Xss428K in jvm-server.options directly? Or we should judge the platform and pagesize in cassandra script, then decide the Xss size dynamically? > Per-thread stack size is too small on aarch64 CentOS > > > Key: CASSANDRA-15446 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15446 > Project: Cassandra > Issue Type: Bug > Components: Local/Config, Local/Startup and Shutdown >Reporter: Heming Fu >Assignee: Heming Fu >Priority: Normal > Fix For: 2.1.x, 2.2.x, 3.0.x, 3.11.x > > > Hi all, > I found an issue when I tried to start cassandra on my aarch64 CentOS7.6, > however no errors on Ubuntu. Of course I could increase -Xss in jvm.options > to fix it, but this issue also caused Cassandra's docker images from docker > hub could not run containers on this OS. > The information of my current environment and root cause of this issue were > shown below. > *Error* > The stack size specified is too small, Specify at least 328k > Error: Could not create the Java Virtual Machine. > Error: A fatal exception has occurred. Program will exit. > *Version* > Cassandra 2.1.21 2.2.15 3.0.19 3.11.5 > *Environment* > $ lscpu > Architecture: aarch64 > Byte Order: Little Endian > $ uname -m > aarch64 > $ java -version > openjdk version "1.8.0_181" > OpenJDK Runtime Environment (build 1.8.0_181-b13) > OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode) > [root@localhost apache-cassandra-3.11.5]# cat /etc/os-release > $ cat /etc/os-release > NAME="CentOS Linux" > VERSION="7 (AltArch)" > ID="centos" > ID_LIKE="rhel fedora" > VERSION_ID="7" > PRETTY_NAME="CentOS Linux 7 (AltArch)" > ANSI_COLOR="0;31" > CPE_NAME="cpe:/o:centos:centos:7" > HOME_URL="https://www.centos.org/"; > BUG_REPORT_URL="https://bugs.centos.org/"; > *Root Cause* > Checked openjdk-1.8.0 source code, the min stack size is calculated by > StackYellowPage, StackRedPage, StackShadowPage, OS page size. Among those > parameters, *default OS page size of aarch64 CentOS 7.6 is 64K, however > aarch64 Ubuntu 18.04 and X86 CentOS are both 4K.* > This difference causes JVM on aarch64 Ubuntu 18.04 needs 164K per-thread > stack size, but 328K required on aarch64 CentOS 7.6. > The formula is > os::Linux::min_stack_allowed = MAX2(os::Linux::min_stack_allowed, > (size_t)(StackYellowPages+StackRedPages+StackShadowPages) * > Linux::page_size() + > (2*BytesPerWord COMPILER2_PRESENT(+1)) * Linux::vm_default_page_size()); > *Parameters on aarch64 CentOS7.6* > intx StackRedPages = 1 > intx StackShadowPages = 1 > intx StackYellowPages = 1 > pageSize 64K > BytesPerWord 8 > vm_default_page_size 8K > As a result, we have min_stack_allowed = (1 + 1 + 1) * 64K + (2 * 8 + 1) * 8K > = 328K > > I could see some similar issues asked for specified achitecture, but no root > cause analyzed. I hope this could help you decide proper stack size for all > common OS. > If you have any suggestion, pls let me know. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16389) Using a cryptographically weak Pseudo Random Number Generator (PRNG)
[ https://issues.apache.org/jira/browse/CASSANDRA-16389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ya Xiao updated CASSANDRA-16389: Description: We are a security research team at Virginia Tech. We are doing an empirical study about the usefulness of the existing security vulnerability detection tools. The following is a reported vulnerability by certain tools. We'll so appreciate it if you can give any feedback on it. *Vulnerability Description* In file [cassandra/src/java/org/apache/cassandra/gms/Gossiper.java|https://github.com/apache/cassandra/blob/79e693e16e2152097c5b27d2d7aaa1763e34f594/src/java/org/apache/cassandra/gms/Gossiper.java], use java.util.Random instead of java.security.SecureRandom at Line 123. *Security Impact:* Java.util.Random is not cryptographically strong and may expose sensitive information to certain types of attacks when used in a security context. *Useful Resources*: [https://cwe.mitre.org/data/definitions/338.html] *Solution we suggest* Replace it with SecureRandom *Please share with us your opinions/comments if there is any* Is the bug report helpful? was: We are a security research team at Virginia Tech. We are doing an empirical study about the usefulness of the existing security vulnerability detection tools. The following is a reported vulnerability by certain tools. We'll so appreciate it if you can give any feedback on it. *Vulnerability Description* In file org.apache.cassandra.gms.Gossiper.java, use java.util.Random instead of java.security.SecureRandom at Line 123. *Security Impact:* Java.util.Random is not cryptographically strong and may expose sensitive information to certain types of attacks when used in a security context. *Useful Resources*: https://cwe.mitre.org/data/definitions/338.html *Solution we suggest* Replace it with SecureRandom *Please share with us your opinions/comments if there is any* Is the bug report helpful? > Using a cryptographically weak Pseudo Random Number Generator (PRNG) > > > Key: CASSANDRA-16389 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16389 > Project: Cassandra > Issue Type: Improvement >Reporter: Ya Xiao >Priority: Normal > > We are a security research team at Virginia Tech. We are doing an empirical > study about the usefulness of the existing security vulnerability detection > tools. The following is a reported vulnerability by certain tools. We'll so > appreciate it if you can give any feedback on it. > *Vulnerability Description* > In file > [cassandra/src/java/org/apache/cassandra/gms/Gossiper.java|https://github.com/apache/cassandra/blob/79e693e16e2152097c5b27d2d7aaa1763e34f594/src/java/org/apache/cassandra/gms/Gossiper.java], > use java.util.Random instead of java.security.SecureRandom at Line 123. > *Security Impact:* > Java.util.Random is not cryptographically strong and may expose sensitive > information to certain types of attacks when used in a security context. > *Useful Resources*: > [https://cwe.mitre.org/data/definitions/338.html] > *Solution we suggest* > Replace it with SecureRandom > *Please share with us your opinions/comments if there is any* > Is the bug report helpful? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16389) Using a cryptographically weak Pseudo Random Number Generator (PRNG)
[ https://issues.apache.org/jira/browse/CASSANDRA-16389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ya Xiao updated CASSANDRA-16389: Summary: Using a cryptographically weak Pseudo Random Number Generator (PRNG) (was: Using a cryptographically weak Pseudo Number Generator (PRNG)) > Using a cryptographically weak Pseudo Random Number Generator (PRNG) > > > Key: CASSANDRA-16389 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16389 > Project: Cassandra > Issue Type: Improvement >Reporter: Ya Xiao >Priority: Normal > > We are a security research team at Virginia Tech. We are doing an empirical > study about the usefulness of the existing security vulnerability detection > tools. The following is a reported vulnerability by certain tools. We'll so > appreciate it if you can give any feedback on it. > *Vulnerability Description* > In file org.apache.cassandra.gms.Gossiper.java, use java.util.Random instead > of java.security.SecureRandom at Line 123. > *Security Impact:* > Java.util.Random is not cryptographically strong and may expose sensitive > information to certain types of attacks when used in a security context. > *Useful Resources*: > https://cwe.mitre.org/data/definitions/338.html > *Solution we suggest* > Replace it with SecureRandom > *Please share with us your opinions/comments if there is any* > Is the bug report helpful? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16389) Using a cryptographically weak Pseudo Number Generator (PRNG)
[ https://issues.apache.org/jira/browse/CASSANDRA-16389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ya Xiao updated CASSANDRA-16389: Summary: Using a cryptographically weak Pseudo Number Generator (PRNG) (was: Using a weak Pseudo Number Generator (PRNG)) > Using a cryptographically weak Pseudo Number Generator (PRNG) > - > > Key: CASSANDRA-16389 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16389 > Project: Cassandra > Issue Type: Improvement >Reporter: Ya Xiao >Priority: Normal > > We are a security research team at Virginia Tech. We are doing an empirical > study about the usefulness of the existing security vulnerability detection > tools. The following is a reported vulnerability by certain tools. We'll so > appreciate it if you can give any feedback on it. > *Vulnerability Description* > In file org.apache.cassandra.gms.Gossiper.java, use java.util.Random instead > of java.security.SecureRandom at Line 123. > *Security Impact:* > Java.util.Random is not cryptographically strong and may expose sensitive > information to certain types of attacks when used in a security context. > *Useful Resources*: > https://cwe.mitre.org/data/definitions/338.html > *Solution we suggest* > Replace it with SecureRandom > *Please share with us your opinions/comments if there is any* > Is the bug report helpful? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-16389) Using a weak Pseudo Number Generator (PRNG)
Ya Xiao created CASSANDRA-16389: --- Summary: Using a weak Pseudo Number Generator (PRNG) Key: CASSANDRA-16389 URL: https://issues.apache.org/jira/browse/CASSANDRA-16389 Project: Cassandra Issue Type: Improvement Reporter: Ya Xiao We are a security research team at Virginia Tech. We are doing an empirical study about the usefulness of the existing security vulnerability detection tools. The following is a reported vulnerability by certain tools. We'll so appreciate it if you can give any feedback on it. *Vulnerability Description* In file org.apache.cassandra.gms.Gossiper.java, use java.util.Random instead of java.security.SecureRandom at Line 123. *Security Impact:* Java.util.Random is not cryptographically strong and may expose sensitive information to certain types of attacks when used in a security context. *Useful Resources*: https://cwe.mitre.org/data/definitions/338.html *Solution we suggest* Replace it with SecureRandom *Please share with us your opinions/comments if there is any* Is the bug report helpful? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16284) Too defensive check when picking sstables for preview repair
[ https://issues.apache.org/jira/browse/CASSANDRA-16284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266370#comment-17266370 ] Ekaterina Dimitrova edited comment on CASSANDRA-16284 at 1/15/21, 11:56 PM: Thanks [~marcuse], I am getting the same failure with a different patch and it doesn't look related to this one. I resubmitted the branch to Jenkins [here |https://ci-cassandra.apache.org/job/Cassandra-devbranch/299/]. +1 on its completion, thank you! was (Author: e.dimitrova): Thanks [~marcuse], I am getting the same failure with a different patch and it doesn't look related to this one. I resubmitted the branch to Jenkins [here |https://jenkins-cm4.apache.org/job/Cassandra-devbranch/292/]. +1 on its completion, thank you! > Too defensive check when picking sstables for preview repair > > > Key: CASSANDRA-16284 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16284 > Project: Cassandra > Issue Type: Bug > Components: Consistency/Repair >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 4.0-beta > > > We fail starting any preview repair if any sstable is marked pending but the > session not being finalized. The current check does not care if the range we > are previewing intersects with the sstable marked pending - this means that > we abort too many preview repairs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16388) Add support for withInstanceInitializer to JVM upgrade dtests.
[ https://issues.apache.org/jira/browse/CASSANDRA-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Meredith updated CASSANDRA-16388: - Change Category: Quality Assurance Complexity: Low Hanging Fruit Status: Open (was: Triage Needed) > Add support for withInstanceInitializer to JVM upgrade dtests. > -- > > Key: CASSANDRA-16388 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16388 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest/java >Reporter: Jon Meredith >Assignee: Jon Meredith >Priority: Normal > > Add the methods to the UpgradeTest builder to allow instance initializers to > be used for upgrade tests as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-16388) Add support for withInstanceInitializer to JVM upgrade dtests.
Jon Meredith created CASSANDRA-16388: Summary: Add support for withInstanceInitializer to JVM upgrade dtests. Key: CASSANDRA-16388 URL: https://issues.apache.org/jira/browse/CASSANDRA-16388 Project: Cassandra Issue Type: Improvement Components: Test/dtest/java Reporter: Jon Meredith Assignee: Jon Meredith Add the methods to the UpgradeTest builder to allow instance initializers to be used for upgrade tests as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16181) 4.0 Quality: Replication Test Audit
[ https://issues.apache.org/jira/browse/CASSANDRA-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266400#comment-17266400 ] Caleb Rackliffe commented on CASSANDRA-16181: - Relating this to CASSANDRA-16387, given whatever that problem is, it's affecting the new upgrade tests here. > 4.0 Quality: Replication Test Audit > --- > > Key: CASSANDRA-16181 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16181 > Project: Cassandra > Issue Type: Task > Components: Test/unit >Reporter: Andres de la Peña >Assignee: Caleb Rackliffe >Priority: Normal > Fix For: 4.0-rc > > Time Spent: 7h 20m > Remaining Estimate: 0h > > This is a subtask of CASSANDRA-15579 focusing on replication. > I think that the main reference dtest for this is > [replication_test.py|https://github.com/apache/cassandra-dtest/blob/master/replication_test.py]. > We should identify which other tests cover this and identify what should be > extended, similarly to what has been done with CASSANDRA-15977. > The doc > [here|https://docs.google.com/document/d/1yPbquhAALIkkTRMmyOv5cceD5N5sPFMB1O4iOd3O7FM/edit?usp=sharing] > describes the existing state of testing around replication. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16387) UpgradeTest sporadically failing on schema updates
[ https://issues.apache.org/jira/browse/CASSANDRA-16387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Caleb Rackliffe updated CASSANDRA-16387: Bug Category: Parent values: Correctness(12982)Level 1 values: Test Failure(12990) Complexity: Normal Discovered By: Unit Test Fix Version/s: 4.0-rc Severity: Normal Status: Open (was: Triage Needed) > UpgradeTest sporadically failing on schema updates > -- > > Key: CASSANDRA-16387 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16387 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest/java >Reporter: Caleb Rackliffe >Priority: Normal > Fix For: 4.0-rc > > > We’ve observed {{UpdateTest}} failing during what appears to be a schema > change: > https://app.circleci.com/pipelines/github/maedhroz/cassandra/192/workflows/ed5305e6-e4f9-420e-9f0a-615746dc/jobs/1068 > It almost looks like the Gossiper can’t find its own endpoint state in the > endpoint state map, and the failure is not consistent, which might suggest a > race. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-16387) UpgradeTest sporadically failing on schema updates
Caleb Rackliffe created CASSANDRA-16387: --- Summary: UpgradeTest sporadically failing on schema updates Key: CASSANDRA-16387 URL: https://issues.apache.org/jira/browse/CASSANDRA-16387 Project: Cassandra Issue Type: Bug Components: Test/dtest/java Reporter: Caleb Rackliffe We’ve observed {{UpdateTest}} failing during what appears to be a schema change: https://app.circleci.com/pipelines/github/maedhroz/cassandra/192/workflows/ed5305e6-e4f9-420e-9f0a-615746dc/jobs/1068 It almost looks like the Gossiper can’t find its own endpoint state in the endpoint state map, and the failure is not consistent, which might suggest a race. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16284) Too defensive check when picking sstables for preview repair
[ https://issues.apache.org/jira/browse/CASSANDRA-16284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266370#comment-17266370 ] Ekaterina Dimitrova commented on CASSANDRA-16284: - Thanks [~marcuse], I am getting the same failure with a different patch and it doesn't look related to this one. I resubmitted the branch to Jenkins [here |https://jenkins-cm4.apache.org/job/Cassandra-devbranch/292/]. +1 on its completion, thank you! > Too defensive check when picking sstables for preview repair > > > Key: CASSANDRA-16284 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16284 > Project: Cassandra > Issue Type: Bug > Components: Consistency/Repair >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 4.0-beta > > > We fail starting any preview repair if any sstable is marked pending but the > session not being finalized. The current check does not care if the range we > are previewing intersects with the sstable marked pending - this means that > we abort too many preview repairs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16201) Reduce amount of allocations during batch statement execution
[ https://issues.apache.org/jira/browse/CASSANDRA-16201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17265653#comment-17265653 ] Yifan Cai edited comment on CASSANDRA-16201 at 1/15/21, 9:47 PM: - +1 on all branches. Rebased the branch to the head of each branch and kicked off CI trunk: [https://app.circleci.com/pipelines/github/yifan-c/cassandra?branch=r%2Fkrummas_marcuse%2F16201-4.0-new] 3.11: [https://app.circleci.com/pipelines/github/yifan-c/cassandra?branch=r%2Fkrummas_marcuse%2F16201-3.11-new] 3.0: [https://app.circleci.com/pipelines/github/yifan-c/cassandra?branch=r%2Fkrummas_marcuse%2F16201-3.0-new] EDIT: update the CI result The test 'testIndexMemtableSwitching - org.apache.cassandra.index.sasi.SASIIndexTest' has been failed. It is tracked already in CASSANDRA-15995 There are bunch of dtest failures. Scanned the failures and they do not look relevant to the change. [https://app.circleci.com/pipelines/github/yifan-c/cassandra/176/workflows/a6c6eaaf-b627-40e6-8af8-851f7a0174c6/jobs/923] There are failures from 2 jvm dtest, 'org.apache.cassandra.distributed.test.SimpleReadWriteTest' and 'org.apache.cassandra.distributed.test.CASTest', all due to 'java.lang.OutOfMemoryError: Java heap space'. I am able to run the test successfully from my local. So the failures do not look relevant. Some flaky tests in the 'CASTest' are tracked in CASSANDRA-16355 and CASSANDRA-16317. was (Author: yifanc): +1 on all branches. Rebased the branch to the head of each branch and kicked off CI trunk: [https://app.circleci.com/pipelines/github/yifan-c/cassandra?branch=r%2Fkrummas_marcuse%2F16201-4.0-new] 3.11: [https://app.circleci.com/pipelines/github/yifan-c/cassandra?branch=r%2Fkrummas_marcuse%2F16201-3.11-new] 3.0: [https://app.circleci.com/pipelines/github/yifan-c/cassandra?branch=r%2Fkrummas_marcuse%2F16201-3.0-new] > Reduce amount of allocations during batch statement execution > - > > Key: CASSANDRA-16201 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16201 > Project: Cassandra > Issue Type: Bug > Components: Local/Other >Reporter: Thomas Steinmaurer >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.0-beta > > Attachments: 16201_jfr_3023_alloc.png, 16201_jfr_3023_obj.png, > 16201_jfr_3118_alloc.png, 16201_jfr_3118_obj.png, 16201_jfr_40b3_alloc.png, > 16201_jfr_40b3_obj.png, screenshot-1.png, screenshot-2.png, screenshot-3.png, > screenshot-4.png > > > In a Cas 2.1 / 3.0 / 3.11 / 4.0b2 comparison test with the same load profile, > we see 4.0b2 going OOM from time to time. According to a heap dump, we have > multiple NTR threads in a 3-digit MB range. > This is likely related to object array pre-allocations at the size of > {{BatchUpdatesCollector.updatedRows}} per {{BTree}} although there is always > only 1 {{BTreeRow}} in the {{BTree}}. > !screenshot-1.png|width=100%! > So it seems we have many, many 20K elemnts pre-allocated object arrays > resulting in a shallow heap of 80K each, although there is only one element > in the array. > This sort of pre-allocation is causing a lot of memory pressure. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-website] branch trunk updated (7da1658 -> f14bf42)
This is an automated email from the ASF dual-hosted git repository. mck pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra-website.git. from 7da1658 Update for new Cassandra release 4.0-beta4 add f14bf42 Blog Cassandra Changelog #3 No new revisions were added by this update. Summary of changes: src/_posts/2021-01-19-cass-changelog_3.markdown | 67 + 1 file changed, 67 insertions(+) create mode 100644 src/_posts/2021-01-19-cass-changelog_3.markdown - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15889) Debian package fails to download on Arm-based hosts
[ https://issues.apache.org/jira/browse/CASSANDRA-15889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266310#comment-17266310 ] Ekaterina Dimitrova commented on CASSANDRA-15889: - I wanted to say thanks to all of you... I love the Cassandra community, look what happened after one question - I went for a day off and so many great things happened only in one day! Let me know if I can be of help with anything! Thanks again all for your passion and dedication to the project! > Debian package fails to download on Arm-based hosts > --- > > Key: CASSANDRA-15889 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15889 > Project: Cassandra > Issue Type: Bug > Components: Build >Reporter: Matt Davis >Assignee: Michael Semb Wever >Priority: Normal > Labels: ARM, debian > Fix For: 2.2.x, 3.0.x, 3.11.x, 4.0-rc > > > Following the first three steps of the [Debian install > process|https://cassandra.apache.org/download/], after an apt-get update > you'll see this line: > {code:bash} > $ sudo apt-get update > ... > N: Skipping acquire of configured file 'main/binary-arm64/Packages' as > repository 'https://downloads.apache.org/cassandra/debian 311x InRelease' > doesn't support architecture 'arm64' > {code} > Checking the [Debian > repo|https://dl.bintray.com/apache/cassandra/dists/311x/main/] confirms there > is no aarch64 variant available. > Should you then attempt to install Cassandra: > {code:bash} > $ sudo apt-get install cassandra > Reading package lists... Done > Building dependency tree > Reading state information... Done > Package cassandra is not available, but is referred to by another package. > This may mean that the package is missing, has been obsoleted, or > is only available from another source > E: Package 'cassandra' has no installation candidate > {code} > Note that there is a workaround available: if you specify "amd64" as the arch > for the source, it downloads and runs on Arm without issue: > {code:bash} > echo "deb [arch=amd64] https://downloads.apache.org/cassandra/debian 311x > main" | sudo tee -a /etc/apt/sources.list.d/cassandra.sources.list > {code} > The Redhat RPM contains a "noarch" arch type, so it will download on any > host. (Cassandra does not use separate binaries/releases for different > architectures, so this seems to be the correct approach, but adding an > aarch64 variant would also suffice.) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15889) Debian package fails to download on Arm-based hosts
[ https://issues.apache.org/jira/browse/CASSANDRA-15889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266306#comment-17266306 ] Michael Semb Wever commented on CASSANDRA-15889: [~mattsplat], we can definitely utilise more arm64 servers, particularly if we want all the tests to run on them. The instructions for adding an agent is [here|https://github.com/apache/cassandra-builds/blob/trunk/ASF-jenkins-agents.txt]. Feel free to email me on m...@apache.org > Debian package fails to download on Arm-based hosts > --- > > Key: CASSANDRA-15889 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15889 > Project: Cassandra > Issue Type: Bug > Components: Build >Reporter: Matt Davis >Assignee: Michael Semb Wever >Priority: Normal > Labels: ARM, debian > Fix For: 2.2.x, 3.0.x, 3.11.x, 4.0-rc > > > Following the first three steps of the [Debian install > process|https://cassandra.apache.org/download/], after an apt-get update > you'll see this line: > {code:bash} > $ sudo apt-get update > ... > N: Skipping acquire of configured file 'main/binary-arm64/Packages' as > repository 'https://downloads.apache.org/cassandra/debian 311x InRelease' > doesn't support architecture 'arm64' > {code} > Checking the [Debian > repo|https://dl.bintray.com/apache/cassandra/dists/311x/main/] confirms there > is no aarch64 variant available. > Should you then attempt to install Cassandra: > {code:bash} > $ sudo apt-get install cassandra > Reading package lists... Done > Building dependency tree > Reading state information... Done > Package cassandra is not available, but is referred to by another package. > This may mean that the package is missing, has been obsoleted, or > is only available from another source > E: Package 'cassandra' has no installation candidate > {code} > Note that there is a workaround available: if you specify "amd64" as the arch > for the source, it downloads and runs on Arm without issue: > {code:bash} > echo "deb [arch=amd64] https://downloads.apache.org/cassandra/debian 311x > main" | sudo tee -a /etc/apt/sources.list.d/cassandra.sources.list > {code} > The Redhat RPM contains a "noarch" arch type, so it will download on any > host. (Cassandra does not use separate binaries/releases for different > architectures, so this seems to be the correct approach, but adding an > aarch64 variant would also suffice.) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16377) Add dtest-upgrade to Jenkins pipeline
[ https://issues.apache.org/jira/browse/CASSANDRA-16377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262557#comment-17262557 ] Michael Semb Wever edited comment on CASSANDRA-16377 at 1/15/21, 7:16 PM: -- WIP - cassandra :: https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/trunk_16377 - cassandra-builds :: https://github.com/apache/cassandra-builds/compare/trunk...thelastpickle:mck/16377 was (Author: michaelsembwever): WIP – https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/trunk_16377 > Add dtest-upgrade to Jenkins pipeline > - > > Key: CASSANDRA-16377 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16377 > Project: Cassandra > Issue Type: Task > Components: CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Low > Fix For: 2.2.x, 3.0.x, 3.11.x, 4.0.x > > > Now that dtest-upgrade test runs are passing they can be considered for > addition to the jenkins pipeline. > https://the-asf.slack.com/archives/CK23JSY2K/p1610132643162700 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16201) Reduce amount of allocations during batch statement execution
[ https://issues.apache.org/jira/browse/CASSANDRA-16201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266226#comment-17266226 ] Benjamin Lerer edited comment on CASSANDRA-16201 at 1/15/21, 5:37 PM: -- [~marcuse] I did not have the time to go through all the branches yet. For the 4.0 branch I got a few comments: * Would it not make sense to use an {{HashMultiset}} rathen than a {{Map}}? according to the [guava documentation|https://github.com/google/guava/wiki/NewCollectionTypesExplained#multiset] they seems to have been developped with that scenario in mind. * In {{BatchStatement.getMutations}}: {code} partitionCounts.computeIfAbsent(stmt.metadata.id, k -> new HashMap<>()); Map perKeyCounts = partitionCounts.get(stmt.metadata.id); {code} Should be: {code} Map perKeyCounts = partitionCounts.computeIfAbsent(stmt.metadata.id, k -> new HashMap<>()); {code} Will it make sense to extract {{k -> new HashMap<>()}} in a variable initialized before the loop ? * Regarding the single table update detection, I wonder if it will not be more efficient to do the comparison on the {{TableId}} rather than the metadata. was (Author: blerer): [~marcuse] I did not have the time to go through all the branches yet. For the 4.0 branch I got a few comments: * Would it not make sense to use an {{HashMultiset}} rathen than a {{Map}}. according to the [guava documentation|https://github.com/google/guava/wiki/NewCollectionTypesExplained#multiset] they seems to have been developped with that scenario in mind. * In {{BatchStatement.getMutations}}: {code} partitionCounts.computeIfAbsent(stmt.metadata.id, k -> new HashMap<>()); Map perKeyCounts = partitionCounts.get(stmt.metadata.id); {code} Should be: {code} Map perKeyCounts = partitionCounts.computeIfAbsent(stmt.metadata.id, k -> new HashMap<>()); {code} Will it make sense to extract {{k -> new HashMap<>()}} in a variable initialized before the loop ? * Regarding the single table update detection, would it not make sense to do the comparison on the {{TableId}} rather than the metadata? > Reduce amount of allocations during batch statement execution > - > > Key: CASSANDRA-16201 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16201 > Project: Cassandra > Issue Type: Bug > Components: Local/Other >Reporter: Thomas Steinmaurer >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.0-beta > > Attachments: 16201_jfr_3023_alloc.png, 16201_jfr_3023_obj.png, > 16201_jfr_3118_alloc.png, 16201_jfr_3118_obj.png, 16201_jfr_40b3_alloc.png, > 16201_jfr_40b3_obj.png, screenshot-1.png, screenshot-2.png, screenshot-3.png, > screenshot-4.png > > > In a Cas 2.1 / 3.0 / 3.11 / 4.0b2 comparison test with the same load profile, > we see 4.0b2 going OOM from time to time. According to a heap dump, we have > multiple NTR threads in a 3-digit MB range. > This is likely related to object array pre-allocations at the size of > {{BatchUpdatesCollector.updatedRows}} per {{BTree}} although there is always > only 1 {{BTreeRow}} in the {{BTree}}. > !screenshot-1.png|width=100%! > So it seems we have many, many 20K elemnts pre-allocated object arrays > resulting in a shallow heap of 80K each, although there is only one element > in the array. > This sort of pre-allocation is causing a lot of memory pressure. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16201) Reduce amount of allocations during batch statement execution
[ https://issues.apache.org/jira/browse/CASSANDRA-16201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266226#comment-17266226 ] Benjamin Lerer commented on CASSANDRA-16201: [~marcuse] I did not have the time to go through all the branches yet. For the 4.0 branch I got a few comments: * Would it not make sense to use an {{HashMultiset}} rathen than a {{Map}}. according to the [guava documentation|https://github.com/google/guava/wiki/NewCollectionTypesExplained#multiset] they seems to have been developped with that scenario in mind. * In {{BatchStatement.getMutations}}: {code} partitionCounts.computeIfAbsent(stmt.metadata.id, k -> new HashMap<>()); Map perKeyCounts = partitionCounts.get(stmt.metadata.id); {code} Should be: {code} Map perKeyCounts = partitionCounts.computeIfAbsent(stmt.metadata.id, k -> new HashMap<>()); {code} Will it make sense to extract {{k -> new HashMap<>()}} in a variable initialized before the loop ? * Regarding the single table update detection, would it not make sense to do the comparison on the {{TableId}} rather than the metadata? > Reduce amount of allocations during batch statement execution > - > > Key: CASSANDRA-16201 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16201 > Project: Cassandra > Issue Type: Bug > Components: Local/Other >Reporter: Thomas Steinmaurer >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.0-beta > > Attachments: 16201_jfr_3023_alloc.png, 16201_jfr_3023_obj.png, > 16201_jfr_3118_alloc.png, 16201_jfr_3118_obj.png, 16201_jfr_40b3_alloc.png, > 16201_jfr_40b3_obj.png, screenshot-1.png, screenshot-2.png, screenshot-3.png, > screenshot-4.png > > > In a Cas 2.1 / 3.0 / 3.11 / 4.0b2 comparison test with the same load profile, > we see 4.0b2 going OOM from time to time. According to a heap dump, we have > multiple NTR threads in a 3-digit MB range. > This is likely related to object array pre-allocations at the size of > {{BatchUpdatesCollector.updatedRows}} per {{BTree}} although there is always > only 1 {{BTreeRow}} in the {{BTree}}. > !screenshot-1.png|width=100%! > So it seems we have many, many 20K elemnts pre-allocated object arrays > resulting in a shallow heap of 80K each, although there is only one element > in the array. > This sort of pre-allocation is causing a lot of memory pressure. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15889) Debian package fails to download on Arm-based hosts
[ https://issues.apache.org/jira/browse/CASSANDRA-15889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266183#comment-17266183 ] Matt Davis commented on CASSANDRA-15889: [~mck] Let me know if you have any trouble getting CI servers, I can help there as well. > Debian package fails to download on Arm-based hosts > --- > > Key: CASSANDRA-15889 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15889 > Project: Cassandra > Issue Type: Bug > Components: Build >Reporter: Matt Davis >Assignee: Michael Semb Wever >Priority: Normal > Labels: ARM, debian > Fix For: 2.2.x, 3.0.x, 3.11.x, 4.0-rc > > > Following the first three steps of the [Debian install > process|https://cassandra.apache.org/download/], after an apt-get update > you'll see this line: > {code:bash} > $ sudo apt-get update > ... > N: Skipping acquire of configured file 'main/binary-arm64/Packages' as > repository 'https://downloads.apache.org/cassandra/debian 311x InRelease' > doesn't support architecture 'arm64' > {code} > Checking the [Debian > repo|https://dl.bintray.com/apache/cassandra/dists/311x/main/] confirms there > is no aarch64 variant available. > Should you then attempt to install Cassandra: > {code:bash} > $ sudo apt-get install cassandra > Reading package lists... Done > Building dependency tree > Reading state information... Done > Package cassandra is not available, but is referred to by another package. > This may mean that the package is missing, has been obsoleted, or > is only available from another source > E: Package 'cassandra' has no installation candidate > {code} > Note that there is a workaround available: if you specify "amd64" as the arch > for the source, it downloads and runs on Arm without issue: > {code:bash} > echo "deb [arch=amd64] https://downloads.apache.org/cassandra/debian 311x > main" | sudo tee -a /etc/apt/sources.list.d/cassandra.sources.list > {code} > The Redhat RPM contains a "noarch" arch type, so it will download on any > host. (Cassandra does not use separate binaries/releases for different > architectures, so this seems to be the correct approach, but adding an > aarch64 variant would also suffice.) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16376) Response headers to OPTION messages always have Stream ID of zero
[ https://issues.apache.org/jira/browse/CASSANDRA-16376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-16376: Impacts: Clients (was: None) > Response headers to OPTION messages always have Stream ID of zero > -- > > Key: CASSANDRA-16376 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16376 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Client >Reporter: Paulius Bankauskas >Priority: Normal > Labels: protocolv5 > Fix For: 4.0-beta5, 4.0 > > Attachments: cass-dump.pcap > > > It seems that streaming behavior has changed since version 4.0-beta2. When a > new connection is made, all responses to the OPTIONS messages always return > with stream id 0 even if requests are made with stream id 1. This causes > connection failures on some clients. > Attached wireshark dump with an example. > > Node version: 4.0-beta4 > Native protocol version: tested with v3 and v4 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16376) Response headers to OPTION messages always have Stream ID of zero
[ https://issues.apache.org/jira/browse/CASSANDRA-16376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-16376: Labels: protocolv5 (was: ) > Response headers to OPTION messages always have Stream ID of zero > -- > > Key: CASSANDRA-16376 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16376 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Client >Reporter: Paulius Bankauskas >Priority: Normal > Labels: protocolv5 > Fix For: 4.0-beta5, 4.0 > > Attachments: cass-dump.pcap > > > It seems that streaming behavior has changed since version 4.0-beta2. When a > new connection is made, all responses to the OPTIONS messages always return > with stream id 0 even if requests are made with stream id 1. This causes > connection failures on some clients. > Attached wireshark dump with an example. > > Node version: 4.0-beta4 > Native protocol version: tested with v3 and v4 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16374) Restore check for consistent native protocol versions for connection
[ https://issues.apache.org/jira/browse/CASSANDRA-16374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-16374: Labels: protocolv5 (was: ) > Restore check for consistent native protocol versions for connection > > > Key: CASSANDRA-16374 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16374 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Client >Reporter: Sam Tunnicliffe >Assignee: Sam Tunnicliffe >Priority: Normal > Labels: protocolv5 > Fix For: 4.0-beta > > > In protocol v4 and earlier, the frame header is checked during > deserialization to ensure that the version matches the one negotiated for the > connection. > The original intention was to remove this version id from the frame (renamed > to envelope in v5) header. However, there is value in keeping this check as > well as the the one for the beta flag, so it remains in the frame/envelope > format. The validation itself however, was removed by some refactoring as > part of CASSANDRA-15299 and should be added back for all protocol versions > before finalizing v5 and cutting a release candidate. The text detailing its > removal also remains in the proposed spec update attached to CASSANDRA-14688, > which also needs to be updated. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16374) Restore check for consistent native protocol versions for connection
[ https://issues.apache.org/jira/browse/CASSANDRA-16374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-16374: Impacts: Clients (was: None) > Restore check for consistent native protocol versions for connection > > > Key: CASSANDRA-16374 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16374 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Client >Reporter: Sam Tunnicliffe >Assignee: Sam Tunnicliffe >Priority: Normal > Fix For: 4.0-beta > > > In protocol v4 and earlier, the frame header is checked during > deserialization to ensure that the version matches the one negotiated for the > connection. > The original intention was to remove this version id from the frame (renamed > to envelope in v5) header. However, there is value in keeping this check as > well as the the one for the beta flag, so it remains in the frame/envelope > format. The validation itself however, was removed by some refactoring as > part of CASSANDRA-15299 and should be added back for all protocol versions > before finalizing v5 and cutting a release candidate. The text detailing its > removal also remains in the proposed spec update attached to CASSANDRA-14688, > which also needs to be updated. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16374) Restore check for consistent native protocol versions for connection
[ https://issues.apache.org/jira/browse/CASSANDRA-16374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-16374: Test and Documentation Plan: New unit test covering all supported protocol versions. Protocol spec to be finalized in CASSANDRA-14688. Status: Patch Available (was: In Progress) Pushed a branch with the fix and new test, CI is running now. ||branch||Circle CI|Apache CI| |[16374-trunk|https://github.com/beobal/cassandra/tree/16374-trunk]|[circle|https://circleci.com/gh/beobal/cassandra?branch=16374-trunk]|[apache|https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/289]| > Restore check for consistent native protocol versions for connection > > > Key: CASSANDRA-16374 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16374 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Client >Reporter: Sam Tunnicliffe >Assignee: Sam Tunnicliffe >Priority: Normal > Fix For: 4.0-beta > > > In protocol v4 and earlier, the frame header is checked during > deserialization to ensure that the version matches the one negotiated for the > connection. > The original intention was to remove this version id from the frame (renamed > to envelope in v5) header. However, there is value in keeping this check as > well as the the one for the beta flag, so it remains in the frame/envelope > format. The validation itself however, was removed by some refactoring as > part of CASSANDRA-15299 and should be added back for all protocol versions > before finalizing v5 and cutting a release candidate. The text detailing its > removal also remains in the proposed spec update attached to CASSANDRA-14688, > which also needs to be updated. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14898) Key cache loading is very slow when there are many SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-14898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266171#comment-17266171 ] Joey Lynch commented on CASSANDRA-14898: Of course! I would be happy to review and merge the patch as well. > Key cache loading is very slow when there are many SSTables > --- > > Key: CASSANDRA-14898 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14898 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths > Environment: AWS i3.2xlarge, 4 physical cores (8 threads), 60GB of > RAM, loading about 8MB of KeyCache with 10k keys in it. >Reporter: Joey Lynch >Assignee: Joey Lynch >Priority: Low > Labels: Performance, low-hanging-fruit > Attachments: key_cache_load_slow.svg > > > While dealing with a production issue today where some 3.0.17 nodes had close > to ~8k sstables on disk due to excessive write pressure, we had a few nodes > crash due to OOM and then they took close to 17 minutes to load the key cache > and recover. This excessive key cache load significantly increased the > duration of the outage (to mitigate we just removed the saved key cache > files). For example here is one example taking 17 minutes to load 10k keys, > or about 10 keys per second (which is ... very slow): > {noformat} > INFO [pool-3-thread-1] 2018-11-15 21:50:21,885 AutoSavingCache.java:190 - > reading saved cache /mnt/data/cassandra/saved_caches/KeyCache-d.db > INFO [pool-3-thread-1] 2018-11-15 22:07:16,490 AutoSavingCache.java:166 - > Completed loading (1014606 ms; 10103 keys) KeyCache cache > {noformat} > I've witnessed similar behavior in the past with large LCS clusters, and > indeed it appears that any time the number of sstables is large, KeyCache > loading takes a _really_ long time. Today I got a flame graph and I believe > that I found the issue and I think it's reasonably easy to fix. From what I > can tell the {{KeyCacheSerializer::deserialize}} [method > |https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/service/CacheService.java#L445] > which is called for every key is linear in the number of sstables due to the > [call|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/service/CacheService.java#L459] > to {{ColumnFamilyStore::getSSTables}} which ends up calling {{View::select}} > [here|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/db/lifecycle/View.java#L139]. > The {{View::select}} call is linear in the number of sstables and causes a > _lot_ of {{HashSet}} > [resizing|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/db/lifecycle/View.java#L139] > when the number of sstables is much greater than 16 (the default size of the > backing {{HashMap}}). > As we see in the attached flamegraph we spend 50% of our CPU time in these > {{getSSTable}} calls, of which 36% is spent adding sstables to the HashSet in > {{View::select}} and 17% is spent just iterating the sstables in the first > place. A full 16% of CPU time is spent _just resizing the HashMap_. Then > another 4% is spend calling {{CacheService::findDesc}} which does [a linear > search|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/service/CacheService.java#L475] > for the sstable generation. > I believe that this affects at least Cassandra 3.0.17 and trunk, and could be > pretty easily fixed by either caching the getSSTables call or at the very > least pre-sizing the {{HashSet}} in {{View::select}} to be the size of the > sstables map. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14898) Key cache loading is very slow when there are many SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-14898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266161#comment-17266161 ] Venkata Harikrishna Nukala commented on CASSANDRA-14898: [~jolynch] can I pick this ticket? I’m working on preparing a patch for this. > Key cache loading is very slow when there are many SSTables > --- > > Key: CASSANDRA-14898 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14898 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths > Environment: AWS i3.2xlarge, 4 physical cores (8 threads), 60GB of > RAM, loading about 8MB of KeyCache with 10k keys in it. >Reporter: Joey Lynch >Assignee: Joey Lynch >Priority: Low > Labels: Performance, low-hanging-fruit > Attachments: key_cache_load_slow.svg > > > While dealing with a production issue today where some 3.0.17 nodes had close > to ~8k sstables on disk due to excessive write pressure, we had a few nodes > crash due to OOM and then they took close to 17 minutes to load the key cache > and recover. This excessive key cache load significantly increased the > duration of the outage (to mitigate we just removed the saved key cache > files). For example here is one example taking 17 minutes to load 10k keys, > or about 10 keys per second (which is ... very slow): > {noformat} > INFO [pool-3-thread-1] 2018-11-15 21:50:21,885 AutoSavingCache.java:190 - > reading saved cache /mnt/data/cassandra/saved_caches/KeyCache-d.db > INFO [pool-3-thread-1] 2018-11-15 22:07:16,490 AutoSavingCache.java:166 - > Completed loading (1014606 ms; 10103 keys) KeyCache cache > {noformat} > I've witnessed similar behavior in the past with large LCS clusters, and > indeed it appears that any time the number of sstables is large, KeyCache > loading takes a _really_ long time. Today I got a flame graph and I believe > that I found the issue and I think it's reasonably easy to fix. From what I > can tell the {{KeyCacheSerializer::deserialize}} [method > |https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/service/CacheService.java#L445] > which is called for every key is linear in the number of sstables due to the > [call|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/service/CacheService.java#L459] > to {{ColumnFamilyStore::getSSTables}} which ends up calling {{View::select}} > [here|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/db/lifecycle/View.java#L139]. > The {{View::select}} call is linear in the number of sstables and causes a > _lot_ of {{HashSet}} > [resizing|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/db/lifecycle/View.java#L139] > when the number of sstables is much greater than 16 (the default size of the > backing {{HashMap}}). > As we see in the attached flamegraph we spend 50% of our CPU time in these > {{getSSTable}} calls, of which 36% is spent adding sstables to the HashSet in > {{View::select}} and 17% is spent just iterating the sstables in the first > place. A full 16% of CPU time is spent _just resizing the HashMap_. Then > another 4% is spend calling {{CacheService::findDesc}} which does [a linear > search|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/service/CacheService.java#L475] > for the sstable generation. > I believe that this affects at least Cassandra 3.0.17 and trunk, and could be > pretty easily fixed by either caching the getSSTables call or at the very > least pre-sizing the {{HashSet}} in {{View::select}} to be the size of the > sstables map. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-builds] branch trunk updated: Bump email for DataStax donated servers
This is an automated email from the ASF dual-hosted git repository. mck pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra-builds.git The following commit(s) were added to refs/heads/trunk by this push: new 83acd49 Bump email for DataStax donated servers 83acd49 is described below commit 83acd49d14e1961678bd7311af06d8a8575dd18d Author: Mick Semb Wever AuthorDate: Fri Jan 15 17:07:34 2021 +0100 Bump email for DataStax donated servers --- ASF-jenkins-agents.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ASF-jenkins-agents.txt b/ASF-jenkins-agents.txt index 8d91bb1..d2d7475 100644 --- a/ASF-jenkins-agents.txt +++ b/ASF-jenkins-agents.txt @@ -105,7 +105,7 @@ cassandra36 - 64.18.213.249 - Ubuntu 18.04 LTS amd64, 32G RAM, 16 core, donated Contacts for system donators, when console hands may be needed by INFRA: - Datastax: Joshua McKenzie + Datastax: Mick Semb Wever Instaclustr: Stefan Miklosovic alternative group list: ad...@instaclustr.com - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16376) Response headers to OPTION messages always have Stream ID of zero
[ https://issues.apache.org/jira/browse/CASSANDRA-16376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-16376: --- Fix Version/s: 4.0 > Response headers to OPTION messages always have Stream ID of zero > -- > > Key: CASSANDRA-16376 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16376 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Client >Reporter: Paulius Bankauskas >Priority: Normal > Fix For: 4.0-beta5, 4.0 > > Attachments: cass-dump.pcap > > > It seems that streaming behavior has changed since version 4.0-beta2. When a > new connection is made, all responses to the OPTIONS messages always return > with stream id 0 even if requests are made with stream id 1. This causes > connection failures on some clients. > Attached wireshark dump with an example. > > Node version: 4.0-beta4 > Native protocol version: tested with v3 and v4 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16318) Memtable heap size is severely underestimated
[ https://issues.apache.org/jira/browse/CASSANDRA-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ekaterina Dimitrova updated CASSANDRA-16318: Authors: Branimir Lambov, Ekaterina Dimitrova (was: Ekaterina Dimitrova) > Memtable heap size is severely underestimated > - > > Key: CASSANDRA-16318 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16318 > Project: Cassandra > Issue Type: Bug > Components: Local/Memtable >Reporter: Branimir Lambov >Assignee: Ekaterina Dimitrova >Priority: Normal > Fix For: 4.0-beta > > Attachments: image-2020-12-09-10-57-21-994.png, > image-2020-12-09-11-01-31-273.png > > > We seem to be estimating the size of the on-heap memtable metadata to be > around half of what it actually is. For example, during a [read benchmark > which writes 1 million single-long > rows|https://github.com/blambov/cassandra/blob/memtable-heap/test/microbench/org/apache/cassandra/test/microbench/instance/ReadTestSmallPartitions.java] > the memtable reports > {code} > 100 ops, 58.174MiB serialized bytes, 385.284MiB (19%) on heap, 0.000KiB > (0%) off-heap > {code} > while a heap dump taken at this point: > !image-2020-12-09-10-57-21-994.png! > lists an usage of about 666MB altogether. > Switching to {{offheap_objects}}, the reported numbers are > {code} > 100 ops, 58.174MiB serialized bytes, 233.650MiB (12%) on heap, 53.406MiB > (3%) off-heap > {code} > while actual heap usage: > !image-2020-12-09-11-01-31-273.png! > is about 442MB. > Looking at the code we definitely are not counting the > {{AtomicBTreePartition.Holder}}, {{EncodingStats}}, liveness and deletion > info objects associated with each partition, and most probably others. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-harry] 02/02: Removed dangling reference to the harry-runner module.
This is an automated email from the ASF dual-hosted git repository. ifesdjeen pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra-harry.git commit 42b4fd0f5c5e747d84631c45f040a0abf80a694f Author: Gianluca Righetto AuthorDate: Sat Jan 9 04:43:43 2021 -0300 Removed dangling reference to the harry-runner module. --- harry-integration/pom.xml | 6 -- 1 file changed, 6 deletions(-) diff --git a/harry-integration/pom.xml b/harry-integration/pom.xml index c27b55d..4695d38 100755 --- a/harry-integration/pom.xml +++ b/harry-integration/pom.xml @@ -41,12 +41,6 @@ org.apache.cassandra -harry-runner -${project.parent.version} - - - -org.apache.cassandra cassandra-dtest-shaded - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-harry] 01/02: Removed dangling reference to the harry-runner module.
This is an automated email from the ASF dual-hosted git repository. ifesdjeen pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra-harry.git commit ca68b85ed157021923892df4ef0c210befde19b2 Author: Gianluca Righetto AuthorDate: Sat Jan 9 04:41:42 2021 -0300 Removed dangling reference to the harry-runner module. --- harry-integration-external/pom.xml | 6 -- 1 file changed, 6 deletions(-) diff --git a/harry-integration-external/pom.xml b/harry-integration-external/pom.xml index 7af3301..8c57e0e 100755 --- a/harry-integration-external/pom.xml +++ b/harry-integration-external/pom.xml @@ -40,12 +40,6 @@ -org.apache.cassandra -harry-runner -${project.parent.version} - - - junit junit test - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-harry] branch trunk updated (41324f5 -> 42b4fd0)
This is an automated email from the ASF dual-hosted git repository. ifesdjeen pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra-harry.git. from 41324f5 Removed references to harry-runner from Dockerfile. new ca68b85 Removed dangling reference to the harry-runner module. new 42b4fd0 Removed dangling reference to the harry-runner module. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: harry-integration-external/pom.xml | 6 -- harry-integration/pom.xml | 6 -- 2 files changed, 12 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-16374) Restore check for consistent native protocol versions for connection
[ https://issues.apache.org/jira/browse/CASSANDRA-16374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe reassigned CASSANDRA-16374: --- Assignee: Sam Tunnicliffe > Restore check for consistent native protocol versions for connection > > > Key: CASSANDRA-16374 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16374 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Client >Reporter: Sam Tunnicliffe >Assignee: Sam Tunnicliffe >Priority: Normal > Fix For: 4.0-beta > > > In protocol v4 and earlier, the frame header is checked during > deserialization to ensure that the version matches the one negotiated for the > connection. > The original intention was to remove this version id from the frame (renamed > to envelope in v5) header. However, there is value in keeping this check as > well as the the one for the beta flag, so it remains in the frame/envelope > format. The validation itself however, was removed by some refactoring as > part of CASSANDRA-15299 and should be added back for all protocol versions > before finalizing v5 and cutting a release candidate. The text detailing its > removal also remains in the proposed spec update attached to CASSANDRA-14688, > which also needs to be updated. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16376) Response headers to OPTION messages always have Stream ID of zero
[ https://issues.apache.org/jira/browse/CASSANDRA-16376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-16376: Fix Version/s: (was: 4.0-beta) 4.0-beta5 Since Version: 4.0-beta4 Source Control Link: https://github.com/apache/cassandra/commit/7e936e7f2c6ccc73d8e3acd31d7050889ec1efbe Resolution: Fixed Status: Resolved (was: Ready to Commit) Thanks [~pauliusb] & [~mck], committed to trunk with the nit addressed & the eclipse warning in `BurnTestUtil` removed. > Response headers to OPTION messages always have Stream ID of zero > -- > > Key: CASSANDRA-16376 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16376 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Client >Reporter: Paulius Bankauskas >Priority: Normal > Fix For: 4.0-beta5 > > Attachments: cass-dump.pcap > > > It seems that streaming behavior has changed since version 4.0-beta2. When a > new connection is made, all responses to the OPTIONS messages always return > with stream id 0 even if requests are made with stream id 1. This causes > connection failures on some clients. > Attached wireshark dump with an example. > > Node version: 4.0-beta4 > Native protocol version: tested with v3 and v4 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16376) Response headers to OPTION messages always have Stream ID of zero
[ https://issues.apache.org/jira/browse/CASSANDRA-16376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-16376: Status: Ready to Commit (was: Review In Progress) > Response headers to OPTION messages always have Stream ID of zero > -- > > Key: CASSANDRA-16376 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16376 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Client >Reporter: Paulius Bankauskas >Priority: Normal > Fix For: 4.0-beta > > Attachments: cass-dump.pcap > > > It seems that streaming behavior has changed since version 4.0-beta2. When a > new connection is made, all responses to the OPTIONS messages always return > with stream id 0 even if requests are made with stream id 1. This causes > connection failures on some clients. > Attached wireshark dump with an example. > > Node version: 4.0-beta4 > Native protocol version: tested with v3 and v4 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated: Initial client handler correctly sets stream id on responses
This is an automated email from the ASF dual-hosted git repository. samt pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git The following commit(s) were added to refs/heads/trunk by this push: new 7e936e7 Initial client handler correctly sets stream id on responses 7e936e7 is described below commit 7e936e7f2c6ccc73d8e3acd31d7050889ec1efbe Author: Sam Tunnicliffe AuthorDate: Wed Jan 13 10:50:17 2021 + Initial client handler correctly sets stream id on responses Patch by Sam Tunnicliffe; reviewed by Mick Semb Wever for CASSANDRA-16376 --- CHANGES.txt| 1 + .../transport/InitialConnectionHandler.java| 1 + .../apache/cassandra/transport/SimpleClient.java | 4 +- .../apache/cassandra/transport/BurnTestUtil.java | 2 +- .../transport/ProtocolNegotiationTest.java | 101 ++--- 5 files changed, 96 insertions(+), 13 deletions(-) diff --git a/CHANGES.txt b/CHANGES.txt index fbb1bd1..47b47d8 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 4.0-beta5 + * Ensure pre-negotiation native protocol responses have correct stream id (CASSANDRA-16376) * Fix check for -Xlog in cassandra-env.sh (CASSANDRA-16279) * SSLFactory should initialize SSLContext before setting protocols (CASSANDRA-16362) * Restore sasi dependencies jflex, snowball-stemmer, and concurrent-trees, in the cassandra-all pom (CASSANDRA-16303) diff --git a/src/java/org/apache/cassandra/transport/InitialConnectionHandler.java b/src/java/org/apache/cassandra/transport/InitialConnectionHandler.java index 70237c5..77e9232 100644 --- a/src/java/org/apache/cassandra/transport/InitialConnectionHandler.java +++ b/src/java/org/apache/cassandra/transport/InitialConnectionHandler.java @@ -89,6 +89,7 @@ public class InitialConnectionHandler extends ByteToMessageDecoder supportedOptions.put(StartupMessage.COMPRESSION, compressions); supportedOptions.put(StartupMessage.PROTOCOL_VERSIONS, ProtocolVersion.supportedVersions()); SupportedMessage supported = new SupportedMessage(supportedOptions); +supported.setStreamId(inbound.header.streamId); outbound = supported.encode(inbound.header.version); ctx.writeAndFlush(outbound); break; diff --git a/src/java/org/apache/cassandra/transport/SimpleClient.java b/src/java/org/apache/cassandra/transport/SimpleClient.java index 43807a8..5ad4c17 100644 --- a/src/java/org/apache/cassandra/transport/SimpleClient.java +++ b/src/java/org/apache/cassandra/transport/SimpleClient.java @@ -25,6 +25,7 @@ import java.util.*; import java.util.concurrent.*; import java.util.concurrent.atomic.AtomicBoolean; +import com.google.common.annotations.VisibleForTesting; import com.google.common.primitives.Ints; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -200,7 +201,8 @@ public class SimpleClient implements Closeable responseHandler.eventHandler = eventHandler; } -protected void establishConnection() throws IOException +@VisibleForTesting +void establishConnection() throws IOException { // Configure the client. bootstrap = new Bootstrap() diff --git a/test/burn/org/apache/cassandra/transport/BurnTestUtil.java b/test/burn/org/apache/cassandra/transport/BurnTestUtil.java index e7798e1..e7bf6b8 100644 --- a/test/burn/org/apache/cassandra/transport/BurnTestUtil.java +++ b/test/burn/org/apache/cassandra/transport/BurnTestUtil.java @@ -62,7 +62,7 @@ public class BurnTestUtil for (int i = 0; i < sizeCaps.columnCountCap; i++) values[i] = bytes(rnd, sizeCaps.valueMinSize, sizeCaps.valueMaxSize); -return new SimpleStatement(Integer.toString(idx), values); +return new SimpleStatement(Integer.toString(idx), (Object[]) values); } public static QueryMessage generateQueryMessage(int idx, SizeCaps sizeCaps) diff --git a/test/unit/org/apache/cassandra/transport/ProtocolNegotiationTest.java b/test/unit/org/apache/cassandra/transport/ProtocolNegotiationTest.java index 54a68bc..f33d8e6 100644 --- a/test/unit/org/apache/cassandra/transport/ProtocolNegotiationTest.java +++ b/test/unit/org/apache/cassandra/transport/ProtocolNegotiationTest.java @@ -18,17 +18,36 @@ package org.apache.cassandra.transport; +import java.io.IOException; +import java.util.Random; + +import com.google.common.collect.ImmutableMap; +import org.junit.Before; import org.junit.BeforeClass; import org.junit.Test; import com.datastax.driver.core.Cluster; -import com.datastax.driver.core.ProtocolVersion; import com.datastax.driver.core.Session; import org.apache.cassandra.cql3.CQLTester; +import org.apache.cassandra.cql3.QueryOptions; +import org.apache.cassandra.cql3.QueryProcessor; +import org.apache.cassandra.transport.messages.OptionsMessage; +i
[jira] [Updated] (CASSANDRA-16360) CRC32 is inefficient on x86
[ https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-16360: Fix Version/s: (was: 4.0-beta) 4.0.x I think this is definitely worthwhile, but unless somebody has the bandwidth to pick it up soon then it's going to have to wait until the next protocol revision. That needn't be a long way off though, we could start work on that in an early 4.0.x. > CRC32 is inefficient on x86 > --- > > Key: CASSANDRA-16360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16360 > Project: Cassandra > Issue Type: Improvement > Components: Messaging/Client >Reporter: Avi Kivity >Priority: Normal > Labels: protocolv5 > Fix For: 4.0.x > > > The client/server protocol specifies CRC24 and CRC32 as the checksum > algorithm (cql_protocol_V5_framing.asc). Those however are expensive to > compute; this affects both the client and the server. > > A better checksum algorithm is CRC32C, which has hardware support on x86 (as > well as other modern architectures). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15889) Debian package fails to download on Arm-based hosts
[ https://issues.apache.org/jira/browse/CASSANDRA-15889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17265828#comment-17265828 ] Michael Semb Wever commented on CASSANDRA-15889: Sent you an email, thanks [~RenhaiZhao]! > Debian package fails to download on Arm-based hosts > --- > > Key: CASSANDRA-15889 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15889 > Project: Cassandra > Issue Type: Bug > Components: Build >Reporter: Matt Davis >Assignee: Michael Semb Wever >Priority: Normal > Labels: ARM, debian > Fix For: 2.2.x, 3.0.x, 3.11.x, 4.0-rc > > > Following the first three steps of the [Debian install > process|https://cassandra.apache.org/download/], after an apt-get update > you'll see this line: > {code:bash} > $ sudo apt-get update > ... > N: Skipping acquire of configured file 'main/binary-arm64/Packages' as > repository 'https://downloads.apache.org/cassandra/debian 311x InRelease' > doesn't support architecture 'arm64' > {code} > Checking the [Debian > repo|https://dl.bintray.com/apache/cassandra/dists/311x/main/] confirms there > is no aarch64 variant available. > Should you then attempt to install Cassandra: > {code:bash} > $ sudo apt-get install cassandra > Reading package lists... Done > Building dependency tree > Reading state information... Done > Package cassandra is not available, but is referred to by another package. > This may mean that the package is missing, has been obsoleted, or > is only available from another source > E: Package 'cassandra' has no installation candidate > {code} > Note that there is a workaround available: if you specify "amd64" as the arch > for the source, it downloads and runs on Arm without issue: > {code:bash} > echo "deb [arch=amd64] https://downloads.apache.org/cassandra/debian 311x > main" | sudo tee -a /etc/apt/sources.list.d/cassandra.sources.list > {code} > The Redhat RPM contains a "noarch" arch type, so it will download on any > host. (Cassandra does not use separate binaries/releases for different > architectures, so this seems to be the correct approach, but adding an > aarch64 variant would also suffice.) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16162) Improve empty hint file handling on startup
[ https://issues.apache.org/jira/browse/CASSANDRA-16162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-16162: Fix Version/s: (was: 4.0-beta) (was: 3.11.x) (was: 3.0.x) 4.0-beta5 3.11.10 3.0.24 Source Control Link: https://github.com/apache/cassandra/commit/f02e53568dbc193b7ac75cc19b0a7751d5514b95 Resolution: Fixed Status: Resolved (was: Ready to Commit) and committed, thanks > Improve empty hint file handling on startup > --- > > Key: CASSANDRA-16162 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16162 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Hints >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Low > Fix For: 3.0.24, 3.11.10, 4.0-beta5 > > > Since CASSANDRA-14080 we handle empty/corrupt hint files on startup, we > should remove empty files and rename corrupt ones to make sure we don't get > the same exception on every startup -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16162) Improve empty hint file handling on startup
[ https://issues.apache.org/jira/browse/CASSANDRA-16162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-16162: Reviewers: Benjamin Lerer, Yifan Cai (was: Benjamin Lerer) > Improve empty hint file handling on startup > --- > > Key: CASSANDRA-16162 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16162 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Hints >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Low > Fix For: 3.0.x, 3.11.x, 4.0-beta > > > Since CASSANDRA-14080 we handle empty/corrupt hint files on startup, we > should remove empty files and rename corrupt ones to make sure we don't get > the same exception on every startup -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16162) Improve empty hint file handling on startup
[ https://issues.apache.org/jira/browse/CASSANDRA-16162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-16162: Status: Ready to Commit (was: Review In Progress) > Improve empty hint file handling on startup > --- > > Key: CASSANDRA-16162 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16162 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Hints >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Low > Fix For: 3.0.x, 3.11.x, 4.0-beta > > > Since CASSANDRA-14080 we handle empty/corrupt hint files on startup, we > should remove empty files and rename corrupt ones to make sure we don't get > the same exception on every startup -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16162) Improve empty hint file handling on startup
[ https://issues.apache.org/jira/browse/CASSANDRA-16162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-16162: Reviewers: Benjamin Lerer (was: Aleksey Yeschenko, Benjamin Lerer) > Improve empty hint file handling on startup > --- > > Key: CASSANDRA-16162 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16162 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Hints >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Low > Fix For: 3.0.x, 3.11.x, 4.0-beta > > > Since CASSANDRA-14080 we handle empty/corrupt hint files on startup, we > should remove empty files and rename corrupt ones to make sure we don't get > the same exception on every startup -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-3.11' into trunk
This is an automated email from the ASF dual-hosted git repository. marcuse pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 4d0daf091812a37c7846efe2a1b8031592bd0207 Merge: 2355c37 228e33d Author: Marcus Eriksson AuthorDate: Fri Jan 15 09:28:01 2021 +0100 Merge branch 'cassandra-3.11' into trunk CHANGES.txt| 1 + .../apache/cassandra/hints/HintsDescriptor.java| 28 +- .../cassandra/hints/HintsDescriptorTest.java | 34 ++ 3 files changed, 56 insertions(+), 7 deletions(-) diff --cc CHANGES.txt index a925e06,c732513..fbb1bd1 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,46 -1,11 +1,47 @@@ -3.11.10 +4.0-beta5 + * Fix check for -Xlog in cassandra-env.sh (CASSANDRA-16279) + * SSLFactory should initialize SSLContext before setting protocols (CASSANDRA-16362) + * Restore sasi dependencies jflex, snowball-stemmer, and concurrent-trees, in the cassandra-all pom (CASSANDRA-16303) * Fix DecimalDeserializer#toString OOM (CASSANDRA-14925) - * Rate limit validation compactions using compaction_throughput_mb_per_sec (CASSANDRA-16161) - * SASI's `max_compaction_flush_memory_in_mb` settings over 100GB revert to default of 1GB (CASSANDRA-16071) +Merged from 3.11: Merged from 3.0: + * Improve empty hint file handling during startup (CASSANDRA-16162) - * Allow empty string in collections with COPY FROM in cqlsh (CASSANDRA-16372) * Fix skipping on pre-3.0 created compact storage sstables due to missing primary key liveness (CASSANDRA-16226) + * Allow empty string in collections with COPY FROM in cqlsh (CASSANDRA-16372 + +4.0-beta4 + * DROP COMPACT STORAGE should invalidate prepared statements still using CompactTableMetadata (CASSANDRA-16361) + * Update default num_tokens to 16 and allocate_tokens_for_local_replication_factor to 3 (CASSANDRA-13701) + * Remove use of String.intern() (CASSANDRA-15810) + * Fix the missing bb position in ByteBufferAccessor.getUnsignedShort (CASSANDRA-16249) + * Make sure OOM errors are rethrown on truncation failure (CASSANDRA-16254) + * Send back client warnings when creating too many tables/keyspaces (CASSANDRA-16309) + * Add dedicated tcp user timeout for streaming connection (CASSANDRA-16143) + * Add generatetokens script for offline token allocation strategy generation (CASSANDRA-16205) + * Remove Windows scripts (CASSANDRA-16171) + * Improve checksumming and compression in protocol V5 (CASSANDRA-15299) + * Optimised repair streaming improvements (CASSANDRA-16274) + * Update jctools dependency to 3.1.0 (CASSANDRA-16255) + * 'SSLEngine closed already' exception on failed outbound connection (CASSANDRA-16277) + * Drain and/or shutdown might throw because of slow messaging service shutdown (CASSANDRA-16276) + * Upgrade JNA to 5.6.0, dropping support for <=glibc-2.6 systems (CASSANDRA-16212) + * Add saved Host IDs to TokenMetadata at startup (CASSANDRA-16246) + * Ensure that CacheMetrics.requests is picked up by the metric reporter (CASSANDRA-16228) + * Add a ratelimiter to snapshot creation and deletion (CASSANDRA-13019) + * Produce consistent tombstone for reads to avoid digest mistmatch (CASSANDRA-15369) + * Fix SSTableloader issue when restoring a table named backups (CASSANDRA-16235) + * Invalid serialized size for responses caused by increasing message time by 1ms which caused extra bytes in size calculation (CASSANDRA-16103) + * Throw BufferOverflowException from DataOutputBuffer for better visibility (CASSANDRA-16214) + * TLS connections to the storage port on a node without server encryption configured causes java.io.IOException accessing missing keystore (CASSANDRA-16144) + * Internode messaging catches OOMs and does not rethrow (CASSANDRA-15214) + * When a table attempts to clean up metrics, it was cleaning up all global table metrics (CASSANDRA-16095) + * Bring back the accepted encryption protocols list as configurable option (CASSANDRA-13325) + * DigestResolver.getData throws AssertionError since dataResponse is null (CASSANDRA-16097) + * Cannot replace_address /X because it doesn't exist in gossip (CASSANDRA-16213) + * cqlsh row_id resets on page boundaries (CASSANDRA-16160) +Merged from 3.11: + * SASI's `max_compaction_flush_memory_in_mb` settings over 100GB revert to default of 1GB (CASSANDRA-16071) +Merged from 3.0: * Extend the exclusion of replica filtering protection to other indices instead of just SASI (CASSANDRA-16311) * Synchronize transaction logs for JBOD (CASSANDRA-16225) * Fix the counting of cells per partition (CASSANDRA-16259) diff --cc test/unit/org/apache/cassandra/hints/HintsDescriptorTest.java index 2d9f972,bab2356..2fad733 --- a/test/unit/org/apache/cassandra/hints/HintsDescriptorTest.java +++ b/test/unit/org/apache/cassandra/hints/HintsDescriptorTest.java @@@ -24,7 -27,7 +27,6 @@@ import java.util.UUID import com.google.co
[cassandra] 01/01: Merge branch 'cassandra-3.0' into cassandra-3.11
This is an automated email from the ASF dual-hosted git repository. marcuse pushed a commit to branch cassandra-3.11 in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 228e33d22b68ba2bf652f4663fac7c508527f304 Merge: 22a0927 f02e535 Author: Marcus Eriksson AuthorDate: Fri Jan 15 09:26:42 2021 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 CHANGES.txt| 1 + .../apache/cassandra/hints/HintsDescriptor.java| 28 - .../cassandra/hints/HintsDescriptorTest.java | 35 ++ 3 files changed, 58 insertions(+), 6 deletions(-) diff --cc CHANGES.txt index 74eef72,ee4cf6e..c732513 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,10 -1,8 +1,11 @@@ -3.0.24: +3.11.10 + * Fix DecimalDeserializer#toString OOM (CASSANDRA-14925) + * Rate limit validation compactions using compaction_throughput_mb_per_sec (CASSANDRA-16161) + * SASI's `max_compaction_flush_memory_in_mb` settings over 100GB revert to default of 1GB (CASSANDRA-16071) +Merged from 3.0: + * Improve empty hint file handling during startup (CASSANDRA-16162) * Allow empty string in collections with COPY FROM in cqlsh (CASSANDRA-16372) * Fix skipping on pre-3.0 created compact storage sstables due to missing primary key liveness (CASSANDRA-16226) - * Fix DecimalDeserializer#toString OOM (CASSANDRA-14925) * Extend the exclusion of replica filtering protection to other indices instead of just SASI (CASSANDRA-16311) * Synchronize transaction logs for JBOD (CASSANDRA-16225) * Fix the counting of cells per partition (CASSANDRA-16259) diff --cc src/java/org/apache/cassandra/hints/HintsDescriptor.java index 00224e2,d1f1116..429d4d4 --- a/src/java/org/apache/cassandra/hints/HintsDescriptor.java +++ b/src/java/org/apache/cassandra/hints/HintsDescriptor.java @@@ -21,15 -21,15 +21,17 @@@ import java.io.DataInput import java.io.IOException; import java.io.RandomAccessFile; import java.nio.charset.StandardCharsets; + import java.nio.file.Files; import java.nio.file.Path; +import java.util.HashMap; import java.util.Map; import java.util.Optional; import java.util.UUID; import java.util.regex.Pattern; import java.util.zip.CRC32; +import javax.crypto.Cipher; + import com.google.common.annotations.VisibleForTesting; import com.google.common.base.MoreObjects; import com.google.common.base.Objects; import com.google.common.collect.ImmutableMap; - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated (2355c37 -> 4d0daf0)
This is an automated email from the ASF dual-hosted git repository. marcuse pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git. from 2355c37 Make relocate dependencies script less manual new f02e535 Improve empty/corrupt hint file handling on startup new 228e33d Merge branch 'cassandra-3.0' into cassandra-3.11 new 4d0daf0 Merge branch 'cassandra-3.11' into trunk The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 1 + .../apache/cassandra/hints/HintsDescriptor.java| 28 +- .../cassandra/hints/HintsDescriptorTest.java | 34 ++ 3 files changed, 56 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-3.0 updated: Improve empty/corrupt hint file handling on startup
This is an automated email from the ASF dual-hosted git repository. marcuse pushed a commit to branch cassandra-3.0 in repository https://gitbox.apache.org/repos/asf/cassandra.git The following commit(s) were added to refs/heads/cassandra-3.0 by this push: new f02e535 Improve empty/corrupt hint file handling on startup f02e535 is described below commit f02e53568dbc193b7ac75cc19b0a7751d5514b95 Author: Marcus Eriksson AuthorDate: Fri Oct 2 08:23:05 2020 +0200 Improve empty/corrupt hint file handling on startup Patch by marcuse; reviewed by Benjamin Lerer and Yifan Cai for CASSANDRA-16162 --- CHANGES.txt| 1 + .../apache/cassandra/hints/HintsDescriptor.java| 28 - .../cassandra/hints/HintsDescriptorTest.java | 35 ++ 3 files changed, 58 insertions(+), 6 deletions(-) diff --git a/CHANGES.txt b/CHANGES.txt index 5cb4f1d..ee4cf6e 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.24: + * Improve empty hint file handling during startup (CASSANDRA-16162) * Allow empty string in collections with COPY FROM in cqlsh (CASSANDRA-16372) * Fix skipping on pre-3.0 created compact storage sstables due to missing primary key liveness (CASSANDRA-16226) * Fix DecimalDeserializer#toString OOM (CASSANDRA-14925) diff --git a/src/java/org/apache/cassandra/hints/HintsDescriptor.java b/src/java/org/apache/cassandra/hints/HintsDescriptor.java index e9e1c30..d1f1116 100644 --- a/src/java/org/apache/cassandra/hints/HintsDescriptor.java +++ b/src/java/org/apache/cassandra/hints/HintsDescriptor.java @@ -21,6 +21,7 @@ import java.io.DataInput; import java.io.IOException; import java.io.RandomAccessFile; import java.nio.charset.StandardCharsets; +import java.nio.file.Files; import java.nio.file.Path; import java.util.Map; import java.util.Optional; @@ -28,6 +29,7 @@ import java.util.UUID; import java.util.regex.Pattern; import java.util.zip.CRC32; +import com.google.common.annotations.VisibleForTesting; import com.google.common.base.MoreObjects; import com.google.common.base.Objects; import com.google.common.collect.ImmutableMap; @@ -148,11 +150,35 @@ final class HintsDescriptor } catch (IOException e) { -logger.error("Failed to deserialize hints descriptor {}", path.toString(), e); +handleDescriptorIOE(e, path); return Optional.empty(); } } +@VisibleForTesting +static void handleDescriptorIOE(IOException e, Path path) +{ +try +{ +if (Files.size(path) > 0) +{ +String newFileName = path.getFileName().toString().replace(".hints", ".corrupt.hints"); +Path target = path.getParent().resolve(newFileName); +logger.error("Failed to deserialize hints descriptor {} - saving file as {}", path.toString(), target, e); +Files.move(path, target); +} +else +{ +logger.warn("Found empty hints file {} on startup, removing", path.toString()); +Files.delete(path); +} +} +catch (IOException ex) +{ +logger.error("Error handling corrupt hints file {}", path.toString(), ex); +} +} + static HintsDescriptor readFromFile(Path path) { try (RandomAccessFile raf = new RandomAccessFile(path.toFile(), "r")) diff --git a/test/unit/org/apache/cassandra/hints/HintsDescriptorTest.java b/test/unit/org/apache/cassandra/hints/HintsDescriptorTest.java index 08487d1..bab2356 100644 --- a/test/unit/org/apache/cassandra/hints/HintsDescriptorTest.java +++ b/test/unit/org/apache/cassandra/hints/HintsDescriptorTest.java @@ -20,11 +20,14 @@ package org.apache.cassandra.hints; import java.io.DataInput; import java.io.File; import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Collections; import java.util.UUID; import com.google.common.collect.ImmutableMap; import com.google.common.io.ByteStreams; -import com.google.common.io.Files; +import org.junit.Assert; import org.junit.Test; import org.apache.cassandra.io.compress.LZ4Compressor; @@ -33,6 +36,8 @@ import org.apache.cassandra.io.util.DataOutputBuffer; import static junit.framework.Assert.assertEquals; import static junit.framework.Assert.assertNotSame; import static junit.framework.Assert.fail; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; public class HintsDescriptorTest { @@ -100,21 +105,41 @@ public class HintsDescriptorTest ImmutableMap parameters = ImmutableMap.of(); HintsDescriptor expected = new HintsDescriptor(hostId, version, timestamp, parameters); -File directory = Files.createTempDir(); +Path directory = Files.createTempDirectory("hints"); try { -try (HintsW
[cassandra] branch cassandra-3.11 updated (22a0927 -> 228e33d)
This is an automated email from the ASF dual-hosted git repository. marcuse pushed a change to branch cassandra-3.11 in repository https://gitbox.apache.org/repos/asf/cassandra.git. from 22a0927 Merge branch 'cassandra-3.0' into cassandra-3.11 new f02e535 Improve empty/corrupt hint file handling on startup new 228e33d Merge branch 'cassandra-3.0' into cassandra-3.11 The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 1 + .../apache/cassandra/hints/HintsDescriptor.java| 28 - .../cassandra/hints/HintsDescriptorTest.java | 35 ++ 3 files changed, 58 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org