[jira] [Commented] (CASSANDRA-15446) Per-thread stack size is too small on aarch64 CentOS

2021-01-15 Thread zhaorenhai (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266524#comment-17266524
 ] 

zhaorenhai commented on CASSANDRA-15446:


What do you think about changing Xss256k to Xss428K in jvm-server.options 
directly?

Or we should judge the platform and pagesize in cassandra script, then decide 
the Xss size dynamically?

 

> Per-thread stack size is too small on aarch64 CentOS
> 
>
> Key: CASSANDRA-15446
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15446
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/Config, Local/Startup and Shutdown
>Reporter: Heming Fu
>Assignee: Heming Fu
>Priority: Normal
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.11.x
>
>
> Hi all,
> I found an issue when I tried to start cassandra on my aarch64 CentOS7.6, 
> however no errors on Ubuntu. Of course I could increase -Xss in jvm.options 
> to fix it, but this issue also caused Cassandra's docker images from docker 
> hub could not run containers on this OS.
> The information of my current environment and root cause of this issue were 
> shown below.
> *Error*
> The stack size specified is too small, Specify at least 328k
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> *Version*
> Cassandra 2.1.21 2.2.15 3.0.19 3.11.5 
> *Environment*
> $ lscpu
> Architecture: aarch64
> Byte Order: Little Endian
> $ uname -m
> aarch64
> $ java -version
> openjdk version "1.8.0_181"
> OpenJDK Runtime Environment (build 1.8.0_181-b13)
> OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
> [root@localhost apache-cassandra-3.11.5]# cat /etc/os-release
> $ cat /etc/os-release
> NAME="CentOS Linux"
> VERSION="7 (AltArch)"
> ID="centos"
> ID_LIKE="rhel fedora"
> VERSION_ID="7"
> PRETTY_NAME="CentOS Linux 7 (AltArch)"
> ANSI_COLOR="0;31"
> CPE_NAME="cpe:/o:centos:centos:7"
> HOME_URL="https://www.centos.org/";
> BUG_REPORT_URL="https://bugs.centos.org/";
> *Root Cause*
> Checked openjdk-1.8.0 source code, the min stack size is calculated by 
> StackYellowPage, StackRedPage, StackShadowPage, OS page size. Among those 
> parameters, *default OS page size of aarch64 CentOS 7.6 is 64K, however 
> aarch64 Ubuntu 18.04 and X86 CentOS are both 4K.*
> This difference causes JVM on aarch64 Ubuntu 18.04 needs 164K per-thread 
> stack size, but 328K required on aarch64 CentOS 7.6.
> The formula is 
> os::Linux::min_stack_allowed = MAX2(os::Linux::min_stack_allowed,
>  (size_t)(StackYellowPages+StackRedPages+StackShadowPages) * 
> Linux::page_size() +
>  (2*BytesPerWord COMPILER2_PRESENT(+1)) * Linux::vm_default_page_size());
> *Parameters on aarch64 CentOS7.6*
> intx StackRedPages = 1 
>  intx StackShadowPages = 1 
>  intx StackYellowPages = 1 
> pageSize 64K
> BytesPerWord 8
> vm_default_page_size 8K
> As a result, we have min_stack_allowed = (1 + 1 + 1) * 64K + (2 * 8 + 1) * 8K 
> = 328K
>  
> I could see some similar issues asked for specified achitecture, but no root 
> cause analyzed. I hope this could help you decide proper stack size for all 
> common OS.
> If you have any suggestion, pls let me know.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-16389) Using a cryptographically weak Pseudo Random Number Generator (PRNG)

2021-01-15 Thread Ya Xiao (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ya Xiao updated CASSANDRA-16389:

Description: 
We are a security research team at Virginia Tech. We are doing an empirical 
study about the usefulness of the existing security vulnerability detection 
tools. The following is a reported vulnerability by certain tools. We'll so 
appreciate it if you can give any feedback on it.

*Vulnerability Description*

In file 
[cassandra/src/java/org/apache/cassandra/gms/Gossiper.java|https://github.com/apache/cassandra/blob/79e693e16e2152097c5b27d2d7aaa1763e34f594/src/java/org/apache/cassandra/gms/Gossiper.java],
 use java.util.Random instead of java.security.SecureRandom at Line 123.

*Security Impact:*

Java.util.Random is not cryptographically strong and may expose sensitive 
information to certain types of attacks when used in a security context.

*Useful Resources*:

[https://cwe.mitre.org/data/definitions/338.html]

*Solution we suggest*

Replace it with SecureRandom

*Please share with us your opinions/comments if there is any*

Is the bug report helpful?

  was:
We are a security research team at Virginia Tech. We are doing an empirical 
study about the usefulness of the existing security vulnerability detection 
tools. The following is a reported vulnerability by certain tools. We'll so 
appreciate it if you can give any feedback on it.

*Vulnerability Description*

In file org.apache.cassandra.gms.Gossiper.java, use java.util.Random instead of 
java.security.SecureRandom at Line 123.

*Security Impact:*

Java.util.Random is not cryptographically strong and may expose sensitive 
information to certain types of attacks when used in a security context.

*Useful Resources*:

https://cwe.mitre.org/data/definitions/338.html

*Solution we suggest*

Replace it with SecureRandom

*Please share with us your opinions/comments if there is any*

Is the bug report helpful?


> Using a cryptographically weak Pseudo Random Number Generator (PRNG)
> 
>
> Key: CASSANDRA-16389
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16389
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Ya Xiao
>Priority: Normal
>
> We are a security research team at Virginia Tech. We are doing an empirical 
> study about the usefulness of the existing security vulnerability detection 
> tools. The following is a reported vulnerability by certain tools. We'll so 
> appreciate it if you can give any feedback on it.
> *Vulnerability Description*
> In file 
> [cassandra/src/java/org/apache/cassandra/gms/Gossiper.java|https://github.com/apache/cassandra/blob/79e693e16e2152097c5b27d2d7aaa1763e34f594/src/java/org/apache/cassandra/gms/Gossiper.java],
>  use java.util.Random instead of java.security.SecureRandom at Line 123.
> *Security Impact:*
> Java.util.Random is not cryptographically strong and may expose sensitive 
> information to certain types of attacks when used in a security context.
> *Useful Resources*:
> [https://cwe.mitre.org/data/definitions/338.html]
> *Solution we suggest*
> Replace it with SecureRandom
> *Please share with us your opinions/comments if there is any*
> Is the bug report helpful?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-16389) Using a cryptographically weak Pseudo Random Number Generator (PRNG)

2021-01-15 Thread Ya Xiao (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ya Xiao updated CASSANDRA-16389:

Summary: Using a cryptographically weak Pseudo Random Number Generator 
(PRNG)  (was: Using a cryptographically weak Pseudo Number Generator (PRNG))

> Using a cryptographically weak Pseudo Random Number Generator (PRNG)
> 
>
> Key: CASSANDRA-16389
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16389
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Ya Xiao
>Priority: Normal
>
> We are a security research team at Virginia Tech. We are doing an empirical 
> study about the usefulness of the existing security vulnerability detection 
> tools. The following is a reported vulnerability by certain tools. We'll so 
> appreciate it if you can give any feedback on it.
> *Vulnerability Description*
> In file org.apache.cassandra.gms.Gossiper.java, use java.util.Random instead 
> of java.security.SecureRandom at Line 123.
> *Security Impact:*
> Java.util.Random is not cryptographically strong and may expose sensitive 
> information to certain types of attacks when used in a security context.
> *Useful Resources*:
> https://cwe.mitre.org/data/definitions/338.html
> *Solution we suggest*
> Replace it with SecureRandom
> *Please share with us your opinions/comments if there is any*
> Is the bug report helpful?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-16389) Using a cryptographically weak Pseudo Number Generator (PRNG)

2021-01-15 Thread Ya Xiao (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ya Xiao updated CASSANDRA-16389:

Summary: Using a cryptographically weak Pseudo Number Generator (PRNG)  
(was: Using a weak Pseudo Number Generator (PRNG))

> Using a cryptographically weak Pseudo Number Generator (PRNG)
> -
>
> Key: CASSANDRA-16389
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16389
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Ya Xiao
>Priority: Normal
>
> We are a security research team at Virginia Tech. We are doing an empirical 
> study about the usefulness of the existing security vulnerability detection 
> tools. The following is a reported vulnerability by certain tools. We'll so 
> appreciate it if you can give any feedback on it.
> *Vulnerability Description*
> In file org.apache.cassandra.gms.Gossiper.java, use java.util.Random instead 
> of java.security.SecureRandom at Line 123.
> *Security Impact:*
> Java.util.Random is not cryptographically strong and may expose sensitive 
> information to certain types of attacks when used in a security context.
> *Useful Resources*:
> https://cwe.mitre.org/data/definitions/338.html
> *Solution we suggest*
> Replace it with SecureRandom
> *Please share with us your opinions/comments if there is any*
> Is the bug report helpful?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Created] (CASSANDRA-16389) Using a weak Pseudo Number Generator (PRNG)

2021-01-15 Thread Ya Xiao (Jira)
Ya Xiao created CASSANDRA-16389:
---

 Summary: Using a weak Pseudo Number Generator (PRNG)
 Key: CASSANDRA-16389
 URL: https://issues.apache.org/jira/browse/CASSANDRA-16389
 Project: Cassandra
  Issue Type: Improvement
Reporter: Ya Xiao


We are a security research team at Virginia Tech. We are doing an empirical 
study about the usefulness of the existing security vulnerability detection 
tools. The following is a reported vulnerability by certain tools. We'll so 
appreciate it if you can give any feedback on it.

*Vulnerability Description*

In file org.apache.cassandra.gms.Gossiper.java, use java.util.Random instead of 
java.security.SecureRandom at Line 123.

*Security Impact:*

Java.util.Random is not cryptographically strong and may expose sensitive 
information to certain types of attacks when used in a security context.

*Useful Resources*:

https://cwe.mitre.org/data/definitions/338.html

*Solution we suggest*

Replace it with SecureRandom

*Please share with us your opinions/comments if there is any*

Is the bug report helpful?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Comment Edited] (CASSANDRA-16284) Too defensive check when picking sstables for preview repair

2021-01-15 Thread Ekaterina Dimitrova (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-16284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266370#comment-17266370
 ] 

Ekaterina Dimitrova edited comment on CASSANDRA-16284 at 1/15/21, 11:56 PM:


Thanks [~marcuse], I am getting the same failure with a different patch and it 
doesn't look related to this one.

I resubmitted the branch to Jenkins [here 
|https://ci-cassandra.apache.org/job/Cassandra-devbranch/299/].

+1 on its completion, thank you!


was (Author: e.dimitrova):
Thanks [~marcuse], I am getting the same failure with a different patch and it 
doesn't look related to this one.

I resubmitted the branch to Jenkins [here 
|https://jenkins-cm4.apache.org/job/Cassandra-devbranch/292/].

+1 on its completion, thank you!

> Too defensive check when picking sstables for preview repair
> 
>
> Key: CASSANDRA-16284
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16284
> Project: Cassandra
>  Issue Type: Bug
>  Components: Consistency/Repair
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 4.0-beta
>
>
> We fail starting any preview repair if any sstable is marked pending but the 
> session not being finalized. The current check does not care if the range we 
> are previewing intersects with the sstable marked pending - this means that 
> we abort too many preview repairs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-16388) Add support for withInstanceInitializer to JVM upgrade dtests.

2021-01-15 Thread Jon Meredith (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Meredith updated CASSANDRA-16388:
-
Change Category: Quality Assurance
 Complexity: Low Hanging Fruit
 Status: Open  (was: Triage Needed)

> Add support for withInstanceInitializer to JVM upgrade dtests.
> --
>
> Key: CASSANDRA-16388
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16388
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Test/dtest/java
>Reporter: Jon Meredith
>Assignee: Jon Meredith
>Priority: Normal
>
> Add the methods to the UpgradeTest builder to allow instance initializers to 
> be used for upgrade tests as well.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Created] (CASSANDRA-16388) Add support for withInstanceInitializer to JVM upgrade dtests.

2021-01-15 Thread Jon Meredith (Jira)
Jon Meredith created CASSANDRA-16388:


 Summary: Add support for withInstanceInitializer to JVM upgrade 
dtests.
 Key: CASSANDRA-16388
 URL: https://issues.apache.org/jira/browse/CASSANDRA-16388
 Project: Cassandra
  Issue Type: Improvement
  Components: Test/dtest/java
Reporter: Jon Meredith
Assignee: Jon Meredith


Add the methods to the UpgradeTest builder to allow instance initializers to be 
used for upgrade tests as well.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Commented] (CASSANDRA-16181) 4.0 Quality: Replication Test Audit

2021-01-15 Thread Caleb Rackliffe (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266400#comment-17266400
 ] 

Caleb Rackliffe commented on CASSANDRA-16181:
-

Relating this to CASSANDRA-16387, given whatever that problem is, it's 
affecting the new upgrade tests here.

> 4.0 Quality: Replication Test Audit
> ---
>
> Key: CASSANDRA-16181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16181
> Project: Cassandra
>  Issue Type: Task
>  Components: Test/unit
>Reporter: Andres de la Peña
>Assignee: Caleb Rackliffe
>Priority: Normal
> Fix For: 4.0-rc
>
>  Time Spent: 7h 20m
>  Remaining Estimate: 0h
>
> This is a subtask of CASSANDRA-15579 focusing on replication.
> I think that the main reference dtest for this is 
> [replication_test.py|https://github.com/apache/cassandra-dtest/blob/master/replication_test.py].
>  We should identify which other tests cover this and identify what should be 
> extended, similarly to what has been done with CASSANDRA-15977.
> The doc 
> [here|https://docs.google.com/document/d/1yPbquhAALIkkTRMmyOv5cceD5N5sPFMB1O4iOd3O7FM/edit?usp=sharing]
>  describes the existing state of testing around replication.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-16387) UpgradeTest sporadically failing on schema updates

2021-01-15 Thread Caleb Rackliffe (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Caleb Rackliffe updated CASSANDRA-16387:

 Bug Category: Parent values: Correctness(12982)Level 1 values: Test 
Failure(12990)
   Complexity: Normal
Discovered By: Unit Test
Fix Version/s: 4.0-rc
 Severity: Normal
   Status: Open  (was: Triage Needed)

> UpgradeTest sporadically failing on schema updates
> --
>
> Key: CASSANDRA-16387
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16387
> Project: Cassandra
>  Issue Type: Bug
>  Components: Test/dtest/java
>Reporter: Caleb Rackliffe
>Priority: Normal
> Fix For: 4.0-rc
>
>
> We’ve observed {{UpdateTest}} failing during what appears to be a schema 
> change:
> https://app.circleci.com/pipelines/github/maedhroz/cassandra/192/workflows/ed5305e6-e4f9-420e-9f0a-615746dc/jobs/1068
> It almost looks like the Gossiper can’t find its own endpoint state in the 
> endpoint state map, and the failure is not consistent, which might suggest a 
> race.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Created] (CASSANDRA-16387) UpgradeTest sporadically failing on schema updates

2021-01-15 Thread Caleb Rackliffe (Jira)
Caleb Rackliffe created CASSANDRA-16387:
---

 Summary: UpgradeTest sporadically failing on schema updates
 Key: CASSANDRA-16387
 URL: https://issues.apache.org/jira/browse/CASSANDRA-16387
 Project: Cassandra
  Issue Type: Bug
  Components: Test/dtest/java
Reporter: Caleb Rackliffe


We’ve observed {{UpdateTest}} failing during what appears to be a schema change:

https://app.circleci.com/pipelines/github/maedhroz/cassandra/192/workflows/ed5305e6-e4f9-420e-9f0a-615746dc/jobs/1068

It almost looks like the Gossiper can’t find its own endpoint state in the 
endpoint state map, and the failure is not consistent, which might suggest a 
race.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Commented] (CASSANDRA-16284) Too defensive check when picking sstables for preview repair

2021-01-15 Thread Ekaterina Dimitrova (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-16284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266370#comment-17266370
 ] 

Ekaterina Dimitrova commented on CASSANDRA-16284:
-

Thanks [~marcuse], I am getting the same failure with a different patch and it 
doesn't look related to this one.

I resubmitted the branch to Jenkins [here 
|https://jenkins-cm4.apache.org/job/Cassandra-devbranch/292/].

+1 on its completion, thank you!

> Too defensive check when picking sstables for preview repair
> 
>
> Key: CASSANDRA-16284
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16284
> Project: Cassandra
>  Issue Type: Bug
>  Components: Consistency/Repair
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 4.0-beta
>
>
> We fail starting any preview repair if any sstable is marked pending but the 
> session not being finalized. The current check does not care if the range we 
> are previewing intersects with the sstable marked pending - this means that 
> we abort too many preview repairs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Comment Edited] (CASSANDRA-16201) Reduce amount of allocations during batch statement execution

2021-01-15 Thread Yifan Cai (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-16201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17265653#comment-17265653
 ] 

Yifan Cai edited comment on CASSANDRA-16201 at 1/15/21, 9:47 PM:
-

+1 on all branches. 

Rebased the branch to the head of each branch and kicked off CI

trunk: 
[https://app.circleci.com/pipelines/github/yifan-c/cassandra?branch=r%2Fkrummas_marcuse%2F16201-4.0-new]

3.11: 
[https://app.circleci.com/pipelines/github/yifan-c/cassandra?branch=r%2Fkrummas_marcuse%2F16201-3.11-new]

3.0: 
[https://app.circleci.com/pipelines/github/yifan-c/cassandra?branch=r%2Fkrummas_marcuse%2F16201-3.0-new]

 

EDIT: update the CI result

The test 'testIndexMemtableSwitching - 
org.apache.cassandra.index.sasi.SASIIndexTest' has been failed. It is tracked 
already in CASSANDRA-15995

There are bunch of dtest failures. Scanned the failures and they do not look 
relevant to the change. 
[https://app.circleci.com/pipelines/github/yifan-c/cassandra/176/workflows/a6c6eaaf-b627-40e6-8af8-851f7a0174c6/jobs/923]

There are failures from 2 jvm dtest, 
'org.apache.cassandra.distributed.test.SimpleReadWriteTest' and 
'org.apache.cassandra.distributed.test.CASTest', all due to 
'java.lang.OutOfMemoryError: Java heap space'. I am able to run the test 
successfully from my local. So the failures do not look relevant. Some flaky 
tests in the 'CASTest' are tracked in CASSANDRA-16355 and CASSANDRA-16317.


was (Author: yifanc):
+1 on all branches. 

Rebased the branch to the head of each branch and kicked off CI

trunk: 
[https://app.circleci.com/pipelines/github/yifan-c/cassandra?branch=r%2Fkrummas_marcuse%2F16201-4.0-new]

3.11: 
[https://app.circleci.com/pipelines/github/yifan-c/cassandra?branch=r%2Fkrummas_marcuse%2F16201-3.11-new]

3.0: 
[https://app.circleci.com/pipelines/github/yifan-c/cassandra?branch=r%2Fkrummas_marcuse%2F16201-3.0-new]

> Reduce amount of allocations during batch statement execution
> -
>
> Key: CASSANDRA-16201
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16201
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/Other
>Reporter: Thomas Steinmaurer
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
> Attachments: 16201_jfr_3023_alloc.png, 16201_jfr_3023_obj.png, 
> 16201_jfr_3118_alloc.png, 16201_jfr_3118_obj.png, 16201_jfr_40b3_alloc.png, 
> 16201_jfr_40b3_obj.png, screenshot-1.png, screenshot-2.png, screenshot-3.png, 
> screenshot-4.png
>
>
> In a Cas 2.1 / 3.0 / 3.11 / 4.0b2 comparison test with the same load profile, 
> we see 4.0b2 going OOM from time to time. According to a heap dump, we have 
> multiple NTR threads in a 3-digit MB range.
> This is likely related to object array pre-allocations at the size of 
> {{BatchUpdatesCollector.updatedRows}} per {{BTree}} although there is always 
> only 1 {{BTreeRow}} in the {{BTree}}.
>  !screenshot-1.png|width=100%! 
> So it seems we have many, many 20K elemnts pre-allocated object arrays 
> resulting in a shallow heap of 80K each, although there is only one element 
> in the array.
> This sort of pre-allocation is causing a lot of memory pressure.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[cassandra-website] branch trunk updated (7da1658 -> f14bf42)

2021-01-15 Thread mck
This is an automated email from the ASF dual-hosted git repository.

mck pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra-website.git.


from 7da1658  Update for new Cassandra release  4.0-beta4
 add f14bf42  Blog Cassandra Changelog #3

No new revisions were added by this update.

Summary of changes:
 src/_posts/2021-01-19-cass-changelog_3.markdown | 67 +
 1 file changed, 67 insertions(+)
 create mode 100644 src/_posts/2021-01-19-cass-changelog_3.markdown


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Commented] (CASSANDRA-15889) Debian package fails to download on Arm-based hosts

2021-01-15 Thread Ekaterina Dimitrova (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-15889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266310#comment-17266310
 ] 

Ekaterina Dimitrova commented on CASSANDRA-15889:
-

I wanted to say thanks to all of you... I love the Cassandra community, look 
what happened after one question - I went for a day off and so many great 
things happened only in one day! Let me know if I can be of help with anything! 
Thanks again all for your passion and dedication to the project!

> Debian package fails to download on Arm-based hosts
> ---
>
> Key: CASSANDRA-15889
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15889
> Project: Cassandra
>  Issue Type: Bug
>  Components: Build
>Reporter: Matt Davis
>Assignee: Michael Semb Wever
>Priority: Normal
>  Labels: ARM, debian
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.0-rc
>
>
> Following the first three steps of the [Debian install 
> process|https://cassandra.apache.org/download/], after an apt-get update 
> you'll see this line:
> {code:bash}
> $ sudo apt-get update
> ...
> N: Skipping acquire of configured file 'main/binary-arm64/Packages' as 
> repository 'https://downloads.apache.org/cassandra/debian 311x InRelease' 
> doesn't support architecture 'arm64'
> {code}
> Checking the [Debian 
> repo|https://dl.bintray.com/apache/cassandra/dists/311x/main/] confirms there 
> is no aarch64 variant available.
> Should you then attempt to install Cassandra:
> {code:bash}
> $ sudo apt-get install cassandra
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Package cassandra is not available, but is referred to by another package.
> This may mean that the package is missing, has been obsoleted, or
> is only available from another source
> E: Package 'cassandra' has no installation candidate
> {code}
> Note that there is a workaround available: if you specify "amd64" as the arch 
> for the source, it downloads and runs on Arm without issue:
> {code:bash}
> echo "deb [arch=amd64] https://downloads.apache.org/cassandra/debian 311x 
> main" | sudo tee -a /etc/apt/sources.list.d/cassandra.sources.list
> {code}
> The Redhat RPM contains a "noarch" arch type, so it will download on any 
> host. (Cassandra does not use separate binaries/releases for different 
> architectures, so this seems to be the correct approach, but adding an 
> aarch64 variant would also suffice.)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Commented] (CASSANDRA-15889) Debian package fails to download on Arm-based hosts

2021-01-15 Thread Michael Semb Wever (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-15889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266306#comment-17266306
 ] 

Michael Semb Wever commented on CASSANDRA-15889:


[~mattsplat], we can definitely utilise more arm64 servers, particularly if we 
want all the tests to run on them. The instructions for adding an agent is 
[here|https://github.com/apache/cassandra-builds/blob/trunk/ASF-jenkins-agents.txt].
 Feel free to email me on m...@apache.org

> Debian package fails to download on Arm-based hosts
> ---
>
> Key: CASSANDRA-15889
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15889
> Project: Cassandra
>  Issue Type: Bug
>  Components: Build
>Reporter: Matt Davis
>Assignee: Michael Semb Wever
>Priority: Normal
>  Labels: ARM, debian
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.0-rc
>
>
> Following the first three steps of the [Debian install 
> process|https://cassandra.apache.org/download/], after an apt-get update 
> you'll see this line:
> {code:bash}
> $ sudo apt-get update
> ...
> N: Skipping acquire of configured file 'main/binary-arm64/Packages' as 
> repository 'https://downloads.apache.org/cassandra/debian 311x InRelease' 
> doesn't support architecture 'arm64'
> {code}
> Checking the [Debian 
> repo|https://dl.bintray.com/apache/cassandra/dists/311x/main/] confirms there 
> is no aarch64 variant available.
> Should you then attempt to install Cassandra:
> {code:bash}
> $ sudo apt-get install cassandra
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Package cassandra is not available, but is referred to by another package.
> This may mean that the package is missing, has been obsoleted, or
> is only available from another source
> E: Package 'cassandra' has no installation candidate
> {code}
> Note that there is a workaround available: if you specify "amd64" as the arch 
> for the source, it downloads and runs on Arm without issue:
> {code:bash}
> echo "deb [arch=amd64] https://downloads.apache.org/cassandra/debian 311x 
> main" | sudo tee -a /etc/apt/sources.list.d/cassandra.sources.list
> {code}
> The Redhat RPM contains a "noarch" arch type, so it will download on any 
> host. (Cassandra does not use separate binaries/releases for different 
> architectures, so this seems to be the correct approach, but adding an 
> aarch64 variant would also suffice.)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Comment Edited] (CASSANDRA-16377) Add dtest-upgrade to Jenkins pipeline

2021-01-15 Thread Michael Semb Wever (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-16377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262557#comment-17262557
 ] 

Michael Semb Wever edited comment on CASSANDRA-16377 at 1/15/21, 7:16 PM:
--

WIP 
- cassandra :: 
https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/trunk_16377
- cassandra-builds :: 
https://github.com/apache/cassandra-builds/compare/trunk...thelastpickle:mck/16377


was (Author: michaelsembwever):
WIP – 
https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/trunk_16377

> Add dtest-upgrade to Jenkins pipeline
> -
>
> Key: CASSANDRA-16377
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16377
> Project: Cassandra
>  Issue Type: Task
>  Components: CI
>Reporter: Michael Semb Wever
>Assignee: Michael Semb Wever
>Priority: Low
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.0.x
>
>
> Now that dtest-upgrade test runs are passing they can be considered for 
> addition to the jenkins pipeline.
> https://the-asf.slack.com/archives/CK23JSY2K/p1610132643162700 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Comment Edited] (CASSANDRA-16201) Reduce amount of allocations during batch statement execution

2021-01-15 Thread Benjamin Lerer (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-16201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266226#comment-17266226
 ] 

Benjamin Lerer edited comment on CASSANDRA-16201 at 1/15/21, 5:37 PM:
--

[~marcuse] I did not have the time to go through all the branches yet.
For the 4.0 branch I got a few comments:
* Would it not make sense to use an {{HashMultiset}} rathen than a 
{{Map}}? according to the [guava 
documentation|https://github.com/google/guava/wiki/NewCollectionTypesExplained#multiset]
 they seems to have been developped with that scenario in mind.
* In {{BatchStatement.getMutations}}:
{code}
partitionCounts.computeIfAbsent(stmt.metadata.id, k -> new 
HashMap<>());
Map perKeyCounts = 
partitionCounts.get(stmt.metadata.id);
{code}
   Should be:
{code}
Map perKeyCounts = 
partitionCounts.computeIfAbsent(stmt.metadata.id, k -> new HashMap<>());
{code}
Will it make sense to extract {{k -> new HashMap<>()}} in a variable 
initialized before the loop ?
*  Regarding the  single table update detection, I wonder if it will not be 
more efficient to do the comparison on the {{TableId}} rather than the metadata.


was (Author: blerer):
[~marcuse] I did not have the time to go through all the branches yet.
For the 4.0 branch I got a few comments:
* Would it not make sense to use an {{HashMultiset}} rathen than a 
{{Map}}. according to the [guava 
documentation|https://github.com/google/guava/wiki/NewCollectionTypesExplained#multiset]
 they seems to have been developped with that scenario in mind.
* In {{BatchStatement.getMutations}}:
{code}
partitionCounts.computeIfAbsent(stmt.metadata.id, k -> new 
HashMap<>());
Map perKeyCounts = 
partitionCounts.get(stmt.metadata.id);
{code}
   Should be:
{code}
Map perKeyCounts = 
partitionCounts.computeIfAbsent(stmt.metadata.id, k -> new HashMap<>());
{code}
Will it make sense to extract {{k -> new HashMap<>()}} in a variable 
initialized before the loop ?
*  Regarding the  single table update detection, would it not make sense to do 
the comparison on the {{TableId}} rather than the metadata?

> Reduce amount of allocations during batch statement execution
> -
>
> Key: CASSANDRA-16201
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16201
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/Other
>Reporter: Thomas Steinmaurer
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
> Attachments: 16201_jfr_3023_alloc.png, 16201_jfr_3023_obj.png, 
> 16201_jfr_3118_alloc.png, 16201_jfr_3118_obj.png, 16201_jfr_40b3_alloc.png, 
> 16201_jfr_40b3_obj.png, screenshot-1.png, screenshot-2.png, screenshot-3.png, 
> screenshot-4.png
>
>
> In a Cas 2.1 / 3.0 / 3.11 / 4.0b2 comparison test with the same load profile, 
> we see 4.0b2 going OOM from time to time. According to a heap dump, we have 
> multiple NTR threads in a 3-digit MB range.
> This is likely related to object array pre-allocations at the size of 
> {{BatchUpdatesCollector.updatedRows}} per {{BTree}} although there is always 
> only 1 {{BTreeRow}} in the {{BTree}}.
>  !screenshot-1.png|width=100%! 
> So it seems we have many, many 20K elemnts pre-allocated object arrays 
> resulting in a shallow heap of 80K each, although there is only one element 
> in the array.
> This sort of pre-allocation is causing a lot of memory pressure.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Commented] (CASSANDRA-16201) Reduce amount of allocations during batch statement execution

2021-01-15 Thread Benjamin Lerer (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-16201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266226#comment-17266226
 ] 

Benjamin Lerer commented on CASSANDRA-16201:


[~marcuse] I did not have the time to go through all the branches yet.
For the 4.0 branch I got a few comments:
* Would it not make sense to use an {{HashMultiset}} rathen than a 
{{Map}}. according to the [guava 
documentation|https://github.com/google/guava/wiki/NewCollectionTypesExplained#multiset]
 they seems to have been developped with that scenario in mind.
* In {{BatchStatement.getMutations}}:
{code}
partitionCounts.computeIfAbsent(stmt.metadata.id, k -> new 
HashMap<>());
Map perKeyCounts = 
partitionCounts.get(stmt.metadata.id);
{code}
   Should be:
{code}
Map perKeyCounts = 
partitionCounts.computeIfAbsent(stmt.metadata.id, k -> new HashMap<>());
{code}
Will it make sense to extract {{k -> new HashMap<>()}} in a variable 
initialized before the loop ?
*  Regarding the  single table update detection, would it not make sense to do 
the comparison on the {{TableId}} rather than the metadata?

> Reduce amount of allocations during batch statement execution
> -
>
> Key: CASSANDRA-16201
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16201
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/Other
>Reporter: Thomas Steinmaurer
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
> Attachments: 16201_jfr_3023_alloc.png, 16201_jfr_3023_obj.png, 
> 16201_jfr_3118_alloc.png, 16201_jfr_3118_obj.png, 16201_jfr_40b3_alloc.png, 
> 16201_jfr_40b3_obj.png, screenshot-1.png, screenshot-2.png, screenshot-3.png, 
> screenshot-4.png
>
>
> In a Cas 2.1 / 3.0 / 3.11 / 4.0b2 comparison test with the same load profile, 
> we see 4.0b2 going OOM from time to time. According to a heap dump, we have 
> multiple NTR threads in a 3-digit MB range.
> This is likely related to object array pre-allocations at the size of 
> {{BatchUpdatesCollector.updatedRows}} per {{BTree}} although there is always 
> only 1 {{BTreeRow}} in the {{BTree}}.
>  !screenshot-1.png|width=100%! 
> So it seems we have many, many 20K elemnts pre-allocated object arrays 
> resulting in a shallow heap of 80K each, although there is only one element 
> in the array.
> This sort of pre-allocation is causing a lot of memory pressure.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Commented] (CASSANDRA-15889) Debian package fails to download on Arm-based hosts

2021-01-15 Thread Matt Davis (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-15889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266183#comment-17266183
 ] 

Matt Davis commented on CASSANDRA-15889:


[~mck] Let me know if you have any trouble getting CI servers, I can help there 
as well.

> Debian package fails to download on Arm-based hosts
> ---
>
> Key: CASSANDRA-15889
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15889
> Project: Cassandra
>  Issue Type: Bug
>  Components: Build
>Reporter: Matt Davis
>Assignee: Michael Semb Wever
>Priority: Normal
>  Labels: ARM, debian
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.0-rc
>
>
> Following the first three steps of the [Debian install 
> process|https://cassandra.apache.org/download/], after an apt-get update 
> you'll see this line:
> {code:bash}
> $ sudo apt-get update
> ...
> N: Skipping acquire of configured file 'main/binary-arm64/Packages' as 
> repository 'https://downloads.apache.org/cassandra/debian 311x InRelease' 
> doesn't support architecture 'arm64'
> {code}
> Checking the [Debian 
> repo|https://dl.bintray.com/apache/cassandra/dists/311x/main/] confirms there 
> is no aarch64 variant available.
> Should you then attempt to install Cassandra:
> {code:bash}
> $ sudo apt-get install cassandra
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Package cassandra is not available, but is referred to by another package.
> This may mean that the package is missing, has been obsoleted, or
> is only available from another source
> E: Package 'cassandra' has no installation candidate
> {code}
> Note that there is a workaround available: if you specify "amd64" as the arch 
> for the source, it downloads and runs on Arm without issue:
> {code:bash}
> echo "deb [arch=amd64] https://downloads.apache.org/cassandra/debian 311x 
> main" | sudo tee -a /etc/apt/sources.list.d/cassandra.sources.list
> {code}
> The Redhat RPM contains a "noarch" arch type, so it will download on any 
> host. (Cassandra does not use separate binaries/releases for different 
> architectures, so this seems to be the correct approach, but adding an 
> aarch64 variant would also suffice.)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-16376) Response headers to OPTION messages always have Stream ID of zero

2021-01-15 Thread Sam Tunnicliffe (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe updated CASSANDRA-16376:

Impacts: Clients  (was: None)

> Response headers to OPTION messages always have Stream ID of zero 
> --
>
> Key: CASSANDRA-16376
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16376
> Project: Cassandra
>  Issue Type: Bug
>  Components: Messaging/Client
>Reporter: Paulius Bankauskas
>Priority: Normal
>  Labels: protocolv5
> Fix For: 4.0-beta5, 4.0
>
> Attachments: cass-dump.pcap
>
>
> It seems that streaming behavior has changed since version 4.0-beta2. When a 
> new connection is made, all responses to the OPTIONS messages always return 
> with stream id 0 even if requests are made with stream id 1. This causes 
> connection failures on some clients. 
> Attached wireshark dump with an example.
>  
> Node version: 4.0-beta4
> Native protocol version: tested with v3 and v4



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-16376) Response headers to OPTION messages always have Stream ID of zero

2021-01-15 Thread Sam Tunnicliffe (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe updated CASSANDRA-16376:

Labels: protocolv5  (was: )

> Response headers to OPTION messages always have Stream ID of zero 
> --
>
> Key: CASSANDRA-16376
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16376
> Project: Cassandra
>  Issue Type: Bug
>  Components: Messaging/Client
>Reporter: Paulius Bankauskas
>Priority: Normal
>  Labels: protocolv5
> Fix For: 4.0-beta5, 4.0
>
> Attachments: cass-dump.pcap
>
>
> It seems that streaming behavior has changed since version 4.0-beta2. When a 
> new connection is made, all responses to the OPTIONS messages always return 
> with stream id 0 even if requests are made with stream id 1. This causes 
> connection failures on some clients. 
> Attached wireshark dump with an example.
>  
> Node version: 4.0-beta4
> Native protocol version: tested with v3 and v4



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-16374) Restore check for consistent native protocol versions for connection

2021-01-15 Thread Sam Tunnicliffe (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe updated CASSANDRA-16374:

Labels: protocolv5  (was: )

> Restore check for consistent native protocol versions for connection
> 
>
> Key: CASSANDRA-16374
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16374
> Project: Cassandra
>  Issue Type: Bug
>  Components: Messaging/Client
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
>Priority: Normal
>  Labels: protocolv5
> Fix For: 4.0-beta
>
>
> In protocol v4 and earlier, the frame header is checked during 
> deserialization to ensure that the version matches the one negotiated for the 
> connection.
> The original intention was to remove this version id from the frame (renamed 
> to envelope in v5) header. However, there is value in keeping this check as 
> well as the the one for the beta flag, so it remains in the frame/envelope 
> format. The validation itself however, was removed by some refactoring as 
> part of CASSANDRA-15299 and should be added back for all protocol versions 
> before finalizing v5 and cutting a release candidate. The text detailing its 
> removal also remains in the proposed spec update attached to CASSANDRA-14688, 
> which also needs to be updated.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-16374) Restore check for consistent native protocol versions for connection

2021-01-15 Thread Sam Tunnicliffe (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe updated CASSANDRA-16374:

Impacts: Clients  (was: None)

> Restore check for consistent native protocol versions for connection
> 
>
> Key: CASSANDRA-16374
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16374
> Project: Cassandra
>  Issue Type: Bug
>  Components: Messaging/Client
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
>Priority: Normal
> Fix For: 4.0-beta
>
>
> In protocol v4 and earlier, the frame header is checked during 
> deserialization to ensure that the version matches the one negotiated for the 
> connection.
> The original intention was to remove this version id from the frame (renamed 
> to envelope in v5) header. However, there is value in keeping this check as 
> well as the the one for the beta flag, so it remains in the frame/envelope 
> format. The validation itself however, was removed by some refactoring as 
> part of CASSANDRA-15299 and should be added back for all protocol versions 
> before finalizing v5 and cutting a release candidate. The text detailing its 
> removal also remains in the proposed spec update attached to CASSANDRA-14688, 
> which also needs to be updated.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-16374) Restore check for consistent native protocol versions for connection

2021-01-15 Thread Sam Tunnicliffe (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe updated CASSANDRA-16374:

Test and Documentation Plan: New unit test covering all supported protocol 
versions. Protocol spec to be finalized in CASSANDRA-14688.
 Status: Patch Available  (was: In Progress)

Pushed a branch with the fix and new test, CI is running now.
||branch||Circle CI|Apache CI|
|[16374-trunk|https://github.com/beobal/cassandra/tree/16374-trunk]|[circle|https://circleci.com/gh/beobal/cassandra?branch=16374-trunk]|[apache|https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/289]|

> Restore check for consistent native protocol versions for connection
> 
>
> Key: CASSANDRA-16374
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16374
> Project: Cassandra
>  Issue Type: Bug
>  Components: Messaging/Client
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
>Priority: Normal
> Fix For: 4.0-beta
>
>
> In protocol v4 and earlier, the frame header is checked during 
> deserialization to ensure that the version matches the one negotiated for the 
> connection.
> The original intention was to remove this version id from the frame (renamed 
> to envelope in v5) header. However, there is value in keeping this check as 
> well as the the one for the beta flag, so it remains in the frame/envelope 
> format. The validation itself however, was removed by some refactoring as 
> part of CASSANDRA-15299 and should be added back for all protocol versions 
> before finalizing v5 and cutting a release candidate. The text detailing its 
> removal also remains in the proposed spec update attached to CASSANDRA-14688, 
> which also needs to be updated.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Commented] (CASSANDRA-14898) Key cache loading is very slow when there are many SSTables

2021-01-15 Thread Joey Lynch (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266171#comment-17266171
 ] 

Joey Lynch commented on CASSANDRA-14898:


Of course! I would be happy to review and merge the patch as well.

> Key cache loading is very slow when there are many SSTables
> ---
>
> Key: CASSANDRA-14898
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14898
> Project: Cassandra
>  Issue Type: Bug
>  Components: Legacy/Local Write-Read Paths
> Environment: AWS i3.2xlarge, 4 physical cores (8 threads), 60GB of 
> RAM, loading about 8MB of KeyCache with 10k keys in it.
>Reporter: Joey Lynch
>Assignee: Joey Lynch
>Priority: Low
>  Labels: Performance, low-hanging-fruit
> Attachments: key_cache_load_slow.svg
>
>
> While dealing with a production issue today where some 3.0.17 nodes had close 
> to ~8k sstables on disk due to excessive write pressure, we had a few nodes 
> crash due to OOM and then they took close to 17 minutes to load the key cache 
> and recover. This excessive key cache load significantly increased the 
> duration of the outage (to mitigate we just removed the saved key cache 
> files). For example here is one example taking 17 minutes to load 10k keys, 
> or about 10 keys per second (which is ... very slow):
> {noformat}
> INFO  [pool-3-thread-1] 2018-11-15 21:50:21,885 AutoSavingCache.java:190 - 
> reading saved cache /mnt/data/cassandra/saved_caches/KeyCache-d.db
> INFO  [pool-3-thread-1] 2018-11-15 22:07:16,490 AutoSavingCache.java:166 - 
> Completed loading (1014606 ms; 10103 keys) KeyCache cache
> {noformat}
> I've witnessed similar behavior in the past with large LCS clusters, and 
> indeed it appears that any time the number of sstables is large, KeyCache 
> loading takes a _really_ long time. Today I got a flame graph and I believe 
> that I found the issue and I think it's reasonably easy to fix. From what I 
> can tell the {{KeyCacheSerializer::deserialize}} [method 
> |https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/service/CacheService.java#L445]
>  which is called for every key is linear in the number of sstables due to the 
> [call|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/service/CacheService.java#L459]
>  to {{ColumnFamilyStore::getSSTables}} which ends up calling {{View::select}} 
> [here|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/db/lifecycle/View.java#L139].
>  The {{View::select}} call is linear in the number of sstables and causes a 
> _lot_ of {{HashSet}} 
> [resizing|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/db/lifecycle/View.java#L139]
>  when the number of sstables is much greater than 16 (the default size of the 
> backing {{HashMap}}).
> As we see in the attached flamegraph we spend 50% of our CPU time in these 
> {{getSSTable}} calls, of which 36% is spent adding sstables to the HashSet in 
> {{View::select}} and 17% is spent just iterating the sstables in the first 
> place. A full 16% of CPU time is spent _just resizing the HashMap_. Then 
> another 4% is spend calling {{CacheService::findDesc}} which does [a linear 
> search|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/service/CacheService.java#L475]
>  for the sstable generation.
> I believe that this affects at least Cassandra 3.0.17 and trunk, and could be 
> pretty easily fixed by either caching the getSSTables call or at the very 
> least pre-sizing the {{HashSet}} in {{View::select}} to be the size of the 
> sstables map.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Commented] (CASSANDRA-14898) Key cache loading is very slow when there are many SSTables

2021-01-15 Thread Venkata Harikrishna Nukala (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266161#comment-17266161
 ] 

Venkata Harikrishna Nukala commented on CASSANDRA-14898:


[~jolynch] can I pick this ticket? I’m working on preparing a patch for this.

> Key cache loading is very slow when there are many SSTables
> ---
>
> Key: CASSANDRA-14898
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14898
> Project: Cassandra
>  Issue Type: Bug
>  Components: Legacy/Local Write-Read Paths
> Environment: AWS i3.2xlarge, 4 physical cores (8 threads), 60GB of 
> RAM, loading about 8MB of KeyCache with 10k keys in it.
>Reporter: Joey Lynch
>Assignee: Joey Lynch
>Priority: Low
>  Labels: Performance, low-hanging-fruit
> Attachments: key_cache_load_slow.svg
>
>
> While dealing with a production issue today where some 3.0.17 nodes had close 
> to ~8k sstables on disk due to excessive write pressure, we had a few nodes 
> crash due to OOM and then they took close to 17 minutes to load the key cache 
> and recover. This excessive key cache load significantly increased the 
> duration of the outage (to mitigate we just removed the saved key cache 
> files). For example here is one example taking 17 minutes to load 10k keys, 
> or about 10 keys per second (which is ... very slow):
> {noformat}
> INFO  [pool-3-thread-1] 2018-11-15 21:50:21,885 AutoSavingCache.java:190 - 
> reading saved cache /mnt/data/cassandra/saved_caches/KeyCache-d.db
> INFO  [pool-3-thread-1] 2018-11-15 22:07:16,490 AutoSavingCache.java:166 - 
> Completed loading (1014606 ms; 10103 keys) KeyCache cache
> {noformat}
> I've witnessed similar behavior in the past with large LCS clusters, and 
> indeed it appears that any time the number of sstables is large, KeyCache 
> loading takes a _really_ long time. Today I got a flame graph and I believe 
> that I found the issue and I think it's reasonably easy to fix. From what I 
> can tell the {{KeyCacheSerializer::deserialize}} [method 
> |https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/service/CacheService.java#L445]
>  which is called for every key is linear in the number of sstables due to the 
> [call|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/service/CacheService.java#L459]
>  to {{ColumnFamilyStore::getSSTables}} which ends up calling {{View::select}} 
> [here|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/db/lifecycle/View.java#L139].
>  The {{View::select}} call is linear in the number of sstables and causes a 
> _lot_ of {{HashSet}} 
> [resizing|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/db/lifecycle/View.java#L139]
>  when the number of sstables is much greater than 16 (the default size of the 
> backing {{HashMap}}).
> As we see in the attached flamegraph we spend 50% of our CPU time in these 
> {{getSSTable}} calls, of which 36% is spent adding sstables to the HashSet in 
> {{View::select}} and 17% is spent just iterating the sstables in the first 
> place. A full 16% of CPU time is spent _just resizing the HashMap_. Then 
> another 4% is spend calling {{CacheService::findDesc}} which does [a linear 
> search|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/service/CacheService.java#L475]
>  for the sstable generation.
> I believe that this affects at least Cassandra 3.0.17 and trunk, and could be 
> pretty easily fixed by either caching the getSSTables call or at the very 
> least pre-sizing the {{HashSet}} in {{View::select}} to be the size of the 
> sstables map.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[cassandra-builds] branch trunk updated: Bump email for DataStax donated servers

2021-01-15 Thread mck
This is an automated email from the ASF dual-hosted git repository.

mck pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra-builds.git


The following commit(s) were added to refs/heads/trunk by this push:
 new 83acd49  Bump email for DataStax donated servers
83acd49 is described below

commit 83acd49d14e1961678bd7311af06d8a8575dd18d
Author: Mick Semb Wever 
AuthorDate: Fri Jan 15 17:07:34 2021 +0100

Bump email for DataStax donated servers
---
 ASF-jenkins-agents.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ASF-jenkins-agents.txt b/ASF-jenkins-agents.txt
index 8d91bb1..d2d7475 100644
--- a/ASF-jenkins-agents.txt
+++ b/ASF-jenkins-agents.txt
@@ -105,7 +105,7 @@ cassandra36 - 64.18.213.249  - Ubuntu 18.04 LTS amd64, 32G 
RAM, 16 core, donated
 
 Contacts for system donators, when console hands may be needed by INFRA:
 
-  Datastax: Joshua McKenzie 
+  Datastax: Mick Semb Wever 
 
   Instaclustr: Stefan Miklosovic 
alternative group list: ad...@instaclustr.com


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-16376) Response headers to OPTION messages always have Stream ID of zero

2021-01-15 Thread Michael Semb Wever (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Semb Wever updated CASSANDRA-16376:
---
Fix Version/s: 4.0

> Response headers to OPTION messages always have Stream ID of zero 
> --
>
> Key: CASSANDRA-16376
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16376
> Project: Cassandra
>  Issue Type: Bug
>  Components: Messaging/Client
>Reporter: Paulius Bankauskas
>Priority: Normal
> Fix For: 4.0-beta5, 4.0
>
> Attachments: cass-dump.pcap
>
>
> It seems that streaming behavior has changed since version 4.0-beta2. When a 
> new connection is made, all responses to the OPTIONS messages always return 
> with stream id 0 even if requests are made with stream id 1. This causes 
> connection failures on some clients. 
> Attached wireshark dump with an example.
>  
> Node version: 4.0-beta4
> Native protocol version: tested with v3 and v4



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-16318) Memtable heap size is severely underestimated

2021-01-15 Thread Ekaterina Dimitrova (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ekaterina Dimitrova updated CASSANDRA-16318:

Authors: Branimir Lambov, Ekaterina Dimitrova  (was: Ekaterina Dimitrova)

> Memtable heap size is severely underestimated
> -
>
> Key: CASSANDRA-16318
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16318
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/Memtable
>Reporter: Branimir Lambov
>Assignee: Ekaterina Dimitrova
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: image-2020-12-09-10-57-21-994.png, 
> image-2020-12-09-11-01-31-273.png
>
>
> We seem to be estimating the size of the on-heap memtable metadata to be 
> around half of what it actually is. For example, during a [read benchmark 
> which writes 1 million single-long 
> rows|https://github.com/blambov/cassandra/blob/memtable-heap/test/microbench/org/apache/cassandra/test/microbench/instance/ReadTestSmallPartitions.java]
>  the memtable reports
> {code}
> 100 ops, 58.174MiB serialized bytes, 385.284MiB (19%) on heap, 0.000KiB 
> (0%) off-heap
> {code}
> while a heap dump taken at this point:
>  !image-2020-12-09-10-57-21-994.png! 
> lists an usage of about 666MB altogether.
> Switching to {{offheap_objects}}, the reported numbers are
> {code}
> 100 ops, 58.174MiB serialized bytes, 233.650MiB (12%) on heap, 53.406MiB 
> (3%) off-heap
> {code}
> while actual heap usage:
>  !image-2020-12-09-11-01-31-273.png! 
> is about 442MB.
> Looking at the code we definitely are not counting the 
> {{AtomicBTreePartition.Holder}}, {{EncodingStats}}, liveness and deletion 
> info objects associated with each partition, and most probably others.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[cassandra-harry] 02/02: Removed dangling reference to the harry-runner module.

2021-01-15 Thread ifesdjeen
This is an automated email from the ASF dual-hosted git repository.

ifesdjeen pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra-harry.git

commit 42b4fd0f5c5e747d84631c45f040a0abf80a694f
Author: Gianluca Righetto 
AuthorDate: Sat Jan 9 04:43:43 2021 -0300

Removed dangling reference to the harry-runner module.
---
 harry-integration/pom.xml | 6 --
 1 file changed, 6 deletions(-)

diff --git a/harry-integration/pom.xml b/harry-integration/pom.xml
index c27b55d..4695d38 100755
--- a/harry-integration/pom.xml
+++ b/harry-integration/pom.xml
@@ -41,12 +41,6 @@
 
 
 org.apache.cassandra
-harry-runner
-${project.parent.version}
-
-
-
-org.apache.cassandra
 cassandra-dtest-shaded
 
 


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[cassandra-harry] 01/02: Removed dangling reference to the harry-runner module.

2021-01-15 Thread ifesdjeen
This is an automated email from the ASF dual-hosted git repository.

ifesdjeen pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra-harry.git

commit ca68b85ed157021923892df4ef0c210befde19b2
Author: Gianluca Righetto 
AuthorDate: Sat Jan 9 04:41:42 2021 -0300

Removed dangling reference to the harry-runner module.
---
 harry-integration-external/pom.xml | 6 --
 1 file changed, 6 deletions(-)

diff --git a/harry-integration-external/pom.xml 
b/harry-integration-external/pom.xml
index 7af3301..8c57e0e 100755
--- a/harry-integration-external/pom.xml
+++ b/harry-integration-external/pom.xml
@@ -40,12 +40,6 @@
 
 
 
-org.apache.cassandra
-harry-runner
-${project.parent.version}
-
-
-
 junit
 junit
 test


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[cassandra-harry] branch trunk updated (41324f5 -> 42b4fd0)

2021-01-15 Thread ifesdjeen
This is an automated email from the ASF dual-hosted git repository.

ifesdjeen pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra-harry.git.


from 41324f5  Removed references to harry-runner from Dockerfile.
 new ca68b85  Removed dangling reference to the harry-runner module.
 new 42b4fd0  Removed dangling reference to the harry-runner module.

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 harry-integration-external/pom.xml | 6 --
 harry-integration/pom.xml  | 6 --
 2 files changed, 12 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Assigned] (CASSANDRA-16374) Restore check for consistent native protocol versions for connection

2021-01-15 Thread Sam Tunnicliffe (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe reassigned CASSANDRA-16374:
---

Assignee: Sam Tunnicliffe

> Restore check for consistent native protocol versions for connection
> 
>
> Key: CASSANDRA-16374
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16374
> Project: Cassandra
>  Issue Type: Bug
>  Components: Messaging/Client
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
>Priority: Normal
> Fix For: 4.0-beta
>
>
> In protocol v4 and earlier, the frame header is checked during 
> deserialization to ensure that the version matches the one negotiated for the 
> connection.
> The original intention was to remove this version id from the frame (renamed 
> to envelope in v5) header. However, there is value in keeping this check as 
> well as the the one for the beta flag, so it remains in the frame/envelope 
> format. The validation itself however, was removed by some refactoring as 
> part of CASSANDRA-15299 and should be added back for all protocol versions 
> before finalizing v5 and cutting a release candidate. The text detailing its 
> removal also remains in the proposed spec update attached to CASSANDRA-14688, 
> which also needs to be updated.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-16376) Response headers to OPTION messages always have Stream ID of zero

2021-01-15 Thread Sam Tunnicliffe (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe updated CASSANDRA-16376:

  Fix Version/s: (was: 4.0-beta)
 4.0-beta5
  Since Version: 4.0-beta4
Source Control Link: 
https://github.com/apache/cassandra/commit/7e936e7f2c6ccc73d8e3acd31d7050889ec1efbe
 Resolution: Fixed
 Status: Resolved  (was: Ready to Commit)

Thanks [~pauliusb] & [~mck], committed to trunk with the nit addressed & the 
eclipse warning in `BurnTestUtil` removed.

> Response headers to OPTION messages always have Stream ID of zero 
> --
>
> Key: CASSANDRA-16376
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16376
> Project: Cassandra
>  Issue Type: Bug
>  Components: Messaging/Client
>Reporter: Paulius Bankauskas
>Priority: Normal
> Fix For: 4.0-beta5
>
> Attachments: cass-dump.pcap
>
>
> It seems that streaming behavior has changed since version 4.0-beta2. When a 
> new connection is made, all responses to the OPTIONS messages always return 
> with stream id 0 even if requests are made with stream id 1. This causes 
> connection failures on some clients. 
> Attached wireshark dump with an example.
>  
> Node version: 4.0-beta4
> Native protocol version: tested with v3 and v4



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-16376) Response headers to OPTION messages always have Stream ID of zero

2021-01-15 Thread Sam Tunnicliffe (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe updated CASSANDRA-16376:

Status: Ready to Commit  (was: Review In Progress)

> Response headers to OPTION messages always have Stream ID of zero 
> --
>
> Key: CASSANDRA-16376
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16376
> Project: Cassandra
>  Issue Type: Bug
>  Components: Messaging/Client
>Reporter: Paulius Bankauskas
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: cass-dump.pcap
>
>
> It seems that streaming behavior has changed since version 4.0-beta2. When a 
> new connection is made, all responses to the OPTIONS messages always return 
> with stream id 0 even if requests are made with stream id 1. This causes 
> connection failures on some clients. 
> Attached wireshark dump with an example.
>  
> Node version: 4.0-beta4
> Native protocol version: tested with v3 and v4



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[cassandra] branch trunk updated: Initial client handler correctly sets stream id on responses

2021-01-15 Thread samt
This is an automated email from the ASF dual-hosted git repository.

samt pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git


The following commit(s) were added to refs/heads/trunk by this push:
 new 7e936e7  Initial client handler correctly sets stream id on responses
7e936e7 is described below

commit 7e936e7f2c6ccc73d8e3acd31d7050889ec1efbe
Author: Sam Tunnicliffe 
AuthorDate: Wed Jan 13 10:50:17 2021 +

Initial client handler correctly sets stream id on responses

Patch by Sam Tunnicliffe; reviewed by Mick Semb Wever for CASSANDRA-16376
---
 CHANGES.txt|   1 +
 .../transport/InitialConnectionHandler.java|   1 +
 .../apache/cassandra/transport/SimpleClient.java   |   4 +-
 .../apache/cassandra/transport/BurnTestUtil.java   |   2 +-
 .../transport/ProtocolNegotiationTest.java | 101 ++---
 5 files changed, 96 insertions(+), 13 deletions(-)

diff --git a/CHANGES.txt b/CHANGES.txt
index fbb1bd1..47b47d8 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 4.0-beta5
+ * Ensure pre-negotiation native protocol responses have correct stream id 
(CASSANDRA-16376)
  * Fix check for -Xlog in cassandra-env.sh (CASSANDRA-16279)
  * SSLFactory should initialize SSLContext before setting protocols 
(CASSANDRA-16362)
  * Restore sasi dependencies jflex, snowball-stemmer, and concurrent-trees, in 
the cassandra-all pom (CASSANDRA-16303)
diff --git 
a/src/java/org/apache/cassandra/transport/InitialConnectionHandler.java 
b/src/java/org/apache/cassandra/transport/InitialConnectionHandler.java
index 70237c5..77e9232 100644
--- a/src/java/org/apache/cassandra/transport/InitialConnectionHandler.java
+++ b/src/java/org/apache/cassandra/transport/InitialConnectionHandler.java
@@ -89,6 +89,7 @@ public class InitialConnectionHandler extends 
ByteToMessageDecoder
 supportedOptions.put(StartupMessage.COMPRESSION, 
compressions);
 supportedOptions.put(StartupMessage.PROTOCOL_VERSIONS, 
ProtocolVersion.supportedVersions());
 SupportedMessage supported = new 
SupportedMessage(supportedOptions);
+supported.setStreamId(inbound.header.streamId);
 outbound = supported.encode(inbound.header.version);
 ctx.writeAndFlush(outbound);
 break;
diff --git a/src/java/org/apache/cassandra/transport/SimpleClient.java 
b/src/java/org/apache/cassandra/transport/SimpleClient.java
index 43807a8..5ad4c17 100644
--- a/src/java/org/apache/cassandra/transport/SimpleClient.java
+++ b/src/java/org/apache/cassandra/transport/SimpleClient.java
@@ -25,6 +25,7 @@ import java.util.*;
 import java.util.concurrent.*;
 import java.util.concurrent.atomic.AtomicBoolean;
 
+import com.google.common.annotations.VisibleForTesting;
 import com.google.common.primitives.Ints;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -200,7 +201,8 @@ public class SimpleClient implements Closeable
 responseHandler.eventHandler = eventHandler;
 }
 
-protected void establishConnection() throws IOException
+@VisibleForTesting
+void establishConnection() throws IOException
 {
 // Configure the client.
 bootstrap = new Bootstrap()
diff --git a/test/burn/org/apache/cassandra/transport/BurnTestUtil.java 
b/test/burn/org/apache/cassandra/transport/BurnTestUtil.java
index e7798e1..e7bf6b8 100644
--- a/test/burn/org/apache/cassandra/transport/BurnTestUtil.java
+++ b/test/burn/org/apache/cassandra/transport/BurnTestUtil.java
@@ -62,7 +62,7 @@ public class BurnTestUtil
 for (int i = 0; i < sizeCaps.columnCountCap; i++)
 values[i] = bytes(rnd, sizeCaps.valueMinSize, 
sizeCaps.valueMaxSize);
 
-return new SimpleStatement(Integer.toString(idx), values);
+return new SimpleStatement(Integer.toString(idx), (Object[]) values);
 }
 
 public static QueryMessage generateQueryMessage(int idx, SizeCaps sizeCaps)
diff --git 
a/test/unit/org/apache/cassandra/transport/ProtocolNegotiationTest.java 
b/test/unit/org/apache/cassandra/transport/ProtocolNegotiationTest.java
index 54a68bc..f33d8e6 100644
--- a/test/unit/org/apache/cassandra/transport/ProtocolNegotiationTest.java
+++ b/test/unit/org/apache/cassandra/transport/ProtocolNegotiationTest.java
@@ -18,17 +18,36 @@
 
 package org.apache.cassandra.transport;
 
+import java.io.IOException;
+import java.util.Random;
+
+import com.google.common.collect.ImmutableMap;
+import org.junit.Before;
 import org.junit.BeforeClass;
 import org.junit.Test;
 
 import com.datastax.driver.core.Cluster;
-import com.datastax.driver.core.ProtocolVersion;
 import com.datastax.driver.core.Session;
 import org.apache.cassandra.cql3.CQLTester;
+import org.apache.cassandra.cql3.QueryOptions;
+import org.apache.cassandra.cql3.QueryProcessor;
+import org.apache.cassandra.transport.messages.OptionsMessage;
+i

[jira] [Updated] (CASSANDRA-16360) CRC32 is inefficient on x86

2021-01-15 Thread Sam Tunnicliffe (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe updated CASSANDRA-16360:

Fix Version/s: (was: 4.0-beta)
   4.0.x

I think this is definitely worthwhile, but unless somebody has the bandwidth to 
pick it up soon then it's going to have to wait until the next protocol 
revision. That needn't be a long way off though, we could start work on that in 
an early 4.0.x. 

> CRC32 is inefficient on x86
> ---
>
> Key: CASSANDRA-16360
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16360
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Messaging/Client
>Reporter: Avi Kivity
>Priority: Normal
>  Labels: protocolv5
> Fix For: 4.0.x
>
>
> The client/server protocol specifies CRC24 and CRC32 as the checksum 
> algorithm (cql_protocol_V5_framing.asc). Those however are expensive to 
> compute; this affects both the client and the server.
>  
> A better checksum algorithm is CRC32C, which has hardware support on x86 (as 
> well as other modern architectures).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Commented] (CASSANDRA-15889) Debian package fails to download on Arm-based hosts

2021-01-15 Thread Michael Semb Wever (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-15889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17265828#comment-17265828
 ] 

Michael Semb Wever commented on CASSANDRA-15889:


Sent you an email, thanks [~RenhaiZhao]!

> Debian package fails to download on Arm-based hosts
> ---
>
> Key: CASSANDRA-15889
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15889
> Project: Cassandra
>  Issue Type: Bug
>  Components: Build
>Reporter: Matt Davis
>Assignee: Michael Semb Wever
>Priority: Normal
>  Labels: ARM, debian
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.0-rc
>
>
> Following the first three steps of the [Debian install 
> process|https://cassandra.apache.org/download/], after an apt-get update 
> you'll see this line:
> {code:bash}
> $ sudo apt-get update
> ...
> N: Skipping acquire of configured file 'main/binary-arm64/Packages' as 
> repository 'https://downloads.apache.org/cassandra/debian 311x InRelease' 
> doesn't support architecture 'arm64'
> {code}
> Checking the [Debian 
> repo|https://dl.bintray.com/apache/cassandra/dists/311x/main/] confirms there 
> is no aarch64 variant available.
> Should you then attempt to install Cassandra:
> {code:bash}
> $ sudo apt-get install cassandra
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Package cassandra is not available, but is referred to by another package.
> This may mean that the package is missing, has been obsoleted, or
> is only available from another source
> E: Package 'cassandra' has no installation candidate
> {code}
> Note that there is a workaround available: if you specify "amd64" as the arch 
> for the source, it downloads and runs on Arm without issue:
> {code:bash}
> echo "deb [arch=amd64] https://downloads.apache.org/cassandra/debian 311x 
> main" | sudo tee -a /etc/apt/sources.list.d/cassandra.sources.list
> {code}
> The Redhat RPM contains a "noarch" arch type, so it will download on any 
> host. (Cassandra does not use separate binaries/releases for different 
> architectures, so this seems to be the correct approach, but adding an 
> aarch64 variant would also suffice.)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-16162) Improve empty hint file handling on startup

2021-01-15 Thread Marcus Eriksson (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-16162:

  Fix Version/s: (was: 4.0-beta)
 (was: 3.11.x)
 (was: 3.0.x)
 4.0-beta5
 3.11.10
 3.0.24
Source Control Link: 
https://github.com/apache/cassandra/commit/f02e53568dbc193b7ac75cc19b0a7751d5514b95
 Resolution: Fixed
 Status: Resolved  (was: Ready to Commit)

and committed, thanks

> Improve empty hint file handling on startup
> ---
>
> Key: CASSANDRA-16162
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16162
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Consistency/Hints
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Low
> Fix For: 3.0.24, 3.11.10, 4.0-beta5
>
>
> Since CASSANDRA-14080 we handle empty/corrupt hint files on startup, we 
> should remove empty files and rename corrupt ones to make sure we don't get 
> the same exception on every startup



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-16162) Improve empty hint file handling on startup

2021-01-15 Thread Marcus Eriksson (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-16162:

Reviewers: Benjamin Lerer, Yifan Cai  (was: Benjamin Lerer)

> Improve empty hint file handling on startup
> ---
>
> Key: CASSANDRA-16162
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16162
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Consistency/Hints
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Low
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
>
> Since CASSANDRA-14080 we handle empty/corrupt hint files on startup, we 
> should remove empty files and rename corrupt ones to make sure we don't get 
> the same exception on every startup



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-16162) Improve empty hint file handling on startup

2021-01-15 Thread Marcus Eriksson (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-16162:

Status: Ready to Commit  (was: Review In Progress)

> Improve empty hint file handling on startup
> ---
>
> Key: CASSANDRA-16162
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16162
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Consistency/Hints
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Low
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
>
> Since CASSANDRA-14080 we handle empty/corrupt hint files on startup, we 
> should remove empty files and rename corrupt ones to make sure we don't get 
> the same exception on every startup



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-16162) Improve empty hint file handling on startup

2021-01-15 Thread Marcus Eriksson (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-16162:

Reviewers: Benjamin Lerer  (was: Aleksey Yeschenko, Benjamin Lerer)

> Improve empty hint file handling on startup
> ---
>
> Key: CASSANDRA-16162
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16162
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Consistency/Hints
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Low
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
>
> Since CASSANDRA-14080 we handle empty/corrupt hint files on startup, we 
> should remove empty files and rename corrupt ones to make sure we don't get 
> the same exception on every startup



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[cassandra] 01/01: Merge branch 'cassandra-3.11' into trunk

2021-01-15 Thread marcuse
This is an automated email from the ASF dual-hosted git repository.

marcuse pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 4d0daf091812a37c7846efe2a1b8031592bd0207
Merge: 2355c37 228e33d
Author: Marcus Eriksson 
AuthorDate: Fri Jan 15 09:28:01 2021 +0100

Merge branch 'cassandra-3.11' into trunk

 CHANGES.txt|  1 +
 .../apache/cassandra/hints/HintsDescriptor.java| 28 +-
 .../cassandra/hints/HintsDescriptorTest.java   | 34 ++
 3 files changed, 56 insertions(+), 7 deletions(-)

diff --cc CHANGES.txt
index a925e06,c732513..fbb1bd1
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,46 -1,11 +1,47 @@@
 -3.11.10
 +4.0-beta5
 + * Fix check for -Xlog in cassandra-env.sh (CASSANDRA-16279)
 + * SSLFactory should initialize SSLContext before setting protocols 
(CASSANDRA-16362)
 + * Restore sasi dependencies jflex, snowball-stemmer, and concurrent-trees, 
in the cassandra-all pom (CASSANDRA-16303)
   * Fix DecimalDeserializer#toString OOM (CASSANDRA-14925)
 - * Rate limit validation compactions using compaction_throughput_mb_per_sec 
(CASSANDRA-16161)
 - * SASI's `max_compaction_flush_memory_in_mb` settings over 100GB revert to 
default of 1GB (CASSANDRA-16071)
 +Merged from 3.11:
  Merged from 3.0:
+  * Improve empty hint file handling during startup (CASSANDRA-16162)
 - * Allow empty string in collections with COPY FROM in cqlsh (CASSANDRA-16372)
   * Fix skipping on pre-3.0 created compact storage sstables due to missing 
primary key liveness (CASSANDRA-16226)
 + * Allow empty string in collections with COPY FROM in cqlsh (CASSANDRA-16372
 +
 +4.0-beta4
 + * DROP COMPACT STORAGE should invalidate prepared statements still using 
CompactTableMetadata (CASSANDRA-16361)
 + * Update default num_tokens to 16 and 
allocate_tokens_for_local_replication_factor to 3 (CASSANDRA-13701)
 + * Remove use of String.intern() (CASSANDRA-15810)
 + * Fix the missing bb position in ByteBufferAccessor.getUnsignedShort 
(CASSANDRA-16249)
 + * Make sure OOM errors are rethrown on truncation failure (CASSANDRA-16254)
 + * Send back client warnings when creating too many tables/keyspaces 
(CASSANDRA-16309)
 + * Add dedicated tcp user timeout for streaming connection (CASSANDRA-16143)
 + * Add generatetokens script for offline token allocation strategy generation 
(CASSANDRA-16205)
 + * Remove Windows scripts (CASSANDRA-16171)
 + * Improve checksumming and compression in protocol V5 (CASSANDRA-15299)
 + * Optimised repair streaming improvements (CASSANDRA-16274)
 + * Update jctools dependency to 3.1.0 (CASSANDRA-16255)
 + * 'SSLEngine closed already' exception on failed outbound connection 
(CASSANDRA-16277)
 + * Drain and/or shutdown might throw because of slow messaging service 
shutdown (CASSANDRA-16276)
 + * Upgrade JNA to 5.6.0, dropping support for <=glibc-2.6 systems 
(CASSANDRA-16212)
 + * Add saved Host IDs to TokenMetadata at startup (CASSANDRA-16246)
 + * Ensure that CacheMetrics.requests is picked up by the metric reporter 
(CASSANDRA-16228)
 + * Add a ratelimiter to snapshot creation and deletion (CASSANDRA-13019)
 + * Produce consistent tombstone for reads to avoid digest mistmatch 
(CASSANDRA-15369)
 + * Fix SSTableloader issue when restoring a table named backups 
(CASSANDRA-16235)
 + * Invalid serialized size for responses caused by increasing message time by 
1ms which caused extra bytes in size calculation (CASSANDRA-16103)
 + * Throw BufferOverflowException from DataOutputBuffer for better visibility 
(CASSANDRA-16214)
 + * TLS connections to the storage port on a node without server encryption 
configured causes java.io.IOException accessing missing keystore 
(CASSANDRA-16144)
 + * Internode messaging catches OOMs and does not rethrow (CASSANDRA-15214)
 + * When a table attempts to clean up metrics, it was cleaning up all global 
table metrics (CASSANDRA-16095)
 + * Bring back the accepted encryption protocols list as configurable option 
(CASSANDRA-13325)
 + * DigestResolver.getData throws AssertionError since dataResponse is null 
(CASSANDRA-16097)
 + * Cannot replace_address /X because it doesn't exist in gossip 
(CASSANDRA-16213)
 + * cqlsh row_id resets on page boundaries (CASSANDRA-16160)
 +Merged from 3.11:
 + * SASI's `max_compaction_flush_memory_in_mb` settings over 100GB revert to 
default of 1GB (CASSANDRA-16071)
 +Merged from 3.0:
   * Extend the exclusion of replica filtering protection to other indices 
instead of just SASI (CASSANDRA-16311)
   * Synchronize transaction logs for JBOD (CASSANDRA-16225)
   * Fix the counting of cells per partition (CASSANDRA-16259)
diff --cc test/unit/org/apache/cassandra/hints/HintsDescriptorTest.java
index 2d9f972,bab2356..2fad733
--- a/test/unit/org/apache/cassandra/hints/HintsDescriptorTest.java
+++ b/test/unit/org/apache/cassandra/hints/HintsDescriptorTest.java
@@@ -24,7 -27,7 +27,6 @@@ import java.util.UUID
  
  import com.google.co

[cassandra] 01/01: Merge branch 'cassandra-3.0' into cassandra-3.11

2021-01-15 Thread marcuse
This is an automated email from the ASF dual-hosted git repository.

marcuse pushed a commit to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 228e33d22b68ba2bf652f4663fac7c508527f304
Merge: 22a0927 f02e535
Author: Marcus Eriksson 
AuthorDate: Fri Jan 15 09:26:42 2021 +0100

Merge branch 'cassandra-3.0' into cassandra-3.11

 CHANGES.txt|  1 +
 .../apache/cassandra/hints/HintsDescriptor.java| 28 -
 .../cassandra/hints/HintsDescriptorTest.java   | 35 ++
 3 files changed, 58 insertions(+), 6 deletions(-)

diff --cc CHANGES.txt
index 74eef72,ee4cf6e..c732513
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,10 -1,8 +1,11 @@@
 -3.0.24:
 +3.11.10
 + * Fix DecimalDeserializer#toString OOM (CASSANDRA-14925)
 + * Rate limit validation compactions using compaction_throughput_mb_per_sec 
(CASSANDRA-16161)
 + * SASI's `max_compaction_flush_memory_in_mb` settings over 100GB revert to 
default of 1GB (CASSANDRA-16071)
 +Merged from 3.0:
+  * Improve empty hint file handling during startup (CASSANDRA-16162)
   * Allow empty string in collections with COPY FROM in cqlsh (CASSANDRA-16372)
   * Fix skipping on pre-3.0 created compact storage sstables due to missing 
primary key liveness (CASSANDRA-16226)
 - * Fix DecimalDeserializer#toString OOM (CASSANDRA-14925)
   * Extend the exclusion of replica filtering protection to other indices 
instead of just SASI (CASSANDRA-16311)
   * Synchronize transaction logs for JBOD (CASSANDRA-16225)
   * Fix the counting of cells per partition (CASSANDRA-16259)
diff --cc src/java/org/apache/cassandra/hints/HintsDescriptor.java
index 00224e2,d1f1116..429d4d4
--- a/src/java/org/apache/cassandra/hints/HintsDescriptor.java
+++ b/src/java/org/apache/cassandra/hints/HintsDescriptor.java
@@@ -21,15 -21,15 +21,17 @@@ import java.io.DataInput
  import java.io.IOException;
  import java.io.RandomAccessFile;
  import java.nio.charset.StandardCharsets;
+ import java.nio.file.Files;
  import java.nio.file.Path;
 +import java.util.HashMap;
  import java.util.Map;
  import java.util.Optional;
  import java.util.UUID;
  import java.util.regex.Pattern;
  import java.util.zip.CRC32;
 +import javax.crypto.Cipher;
  
+ import com.google.common.annotations.VisibleForTesting;
  import com.google.common.base.MoreObjects;
  import com.google.common.base.Objects;
  import com.google.common.collect.ImmutableMap;


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[cassandra] branch trunk updated (2355c37 -> 4d0daf0)

2021-01-15 Thread marcuse
This is an automated email from the ASF dual-hosted git repository.

marcuse pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from 2355c37  Make relocate dependencies script less manual
 new f02e535  Improve empty/corrupt hint file handling on startup
 new 228e33d  Merge branch 'cassandra-3.0' into cassandra-3.11
 new 4d0daf0  Merge branch 'cassandra-3.11' into trunk

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|  1 +
 .../apache/cassandra/hints/HintsDescriptor.java| 28 +-
 .../cassandra/hints/HintsDescriptorTest.java   | 34 ++
 3 files changed, 56 insertions(+), 7 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[cassandra] branch cassandra-3.0 updated: Improve empty/corrupt hint file handling on startup

2021-01-15 Thread marcuse
This is an automated email from the ASF dual-hosted git repository.

marcuse pushed a commit to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git


The following commit(s) were added to refs/heads/cassandra-3.0 by this push:
 new f02e535  Improve empty/corrupt hint file handling on startup
f02e535 is described below

commit f02e53568dbc193b7ac75cc19b0a7751d5514b95
Author: Marcus Eriksson 
AuthorDate: Fri Oct 2 08:23:05 2020 +0200

Improve empty/corrupt hint file handling on startup

Patch by marcuse; reviewed by Benjamin Lerer and Yifan Cai for 
CASSANDRA-16162
---
 CHANGES.txt|  1 +
 .../apache/cassandra/hints/HintsDescriptor.java| 28 -
 .../cassandra/hints/HintsDescriptorTest.java   | 35 ++
 3 files changed, 58 insertions(+), 6 deletions(-)

diff --git a/CHANGES.txt b/CHANGES.txt
index 5cb4f1d..ee4cf6e 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.24:
+ * Improve empty hint file handling during startup (CASSANDRA-16162)
  * Allow empty string in collections with COPY FROM in cqlsh (CASSANDRA-16372)
  * Fix skipping on pre-3.0 created compact storage sstables due to missing 
primary key liveness (CASSANDRA-16226)
  * Fix DecimalDeserializer#toString OOM (CASSANDRA-14925)
diff --git a/src/java/org/apache/cassandra/hints/HintsDescriptor.java 
b/src/java/org/apache/cassandra/hints/HintsDescriptor.java
index e9e1c30..d1f1116 100644
--- a/src/java/org/apache/cassandra/hints/HintsDescriptor.java
+++ b/src/java/org/apache/cassandra/hints/HintsDescriptor.java
@@ -21,6 +21,7 @@ import java.io.DataInput;
 import java.io.IOException;
 import java.io.RandomAccessFile;
 import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
 import java.nio.file.Path;
 import java.util.Map;
 import java.util.Optional;
@@ -28,6 +29,7 @@ import java.util.UUID;
 import java.util.regex.Pattern;
 import java.util.zip.CRC32;
 
+import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.MoreObjects;
 import com.google.common.base.Objects;
 import com.google.common.collect.ImmutableMap;
@@ -148,11 +150,35 @@ final class HintsDescriptor
 }
 catch (IOException e)
 {
-logger.error("Failed to deserialize hints descriptor {}", 
path.toString(), e);
+handleDescriptorIOE(e, path);
 return Optional.empty();
 }
 }
 
+@VisibleForTesting
+static void handleDescriptorIOE(IOException e, Path path)
+{
+try
+{
+if (Files.size(path) > 0)
+{
+String newFileName = 
path.getFileName().toString().replace(".hints", ".corrupt.hints");
+Path target = path.getParent().resolve(newFileName);
+logger.error("Failed to deserialize hints descriptor {} - 
saving file as {}", path.toString(), target, e);
+Files.move(path, target);
+}
+else
+{
+logger.warn("Found empty hints file {} on startup, removing", 
path.toString());
+Files.delete(path);
+}
+}
+catch (IOException ex)
+{
+logger.error("Error handling corrupt hints file {}", 
path.toString(), ex);
+}
+}
+
 static HintsDescriptor readFromFile(Path path)
 {
 try (RandomAccessFile raf = new RandomAccessFile(path.toFile(), "r"))
diff --git a/test/unit/org/apache/cassandra/hints/HintsDescriptorTest.java 
b/test/unit/org/apache/cassandra/hints/HintsDescriptorTest.java
index 08487d1..bab2356 100644
--- a/test/unit/org/apache/cassandra/hints/HintsDescriptorTest.java
+++ b/test/unit/org/apache/cassandra/hints/HintsDescriptorTest.java
@@ -20,11 +20,14 @@ package org.apache.cassandra.hints;
 import java.io.DataInput;
 import java.io.File;
 import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
 import java.util.UUID;
 
 import com.google.common.collect.ImmutableMap;
 import com.google.common.io.ByteStreams;
-import com.google.common.io.Files;
+import org.junit.Assert;
 import org.junit.Test;
 
 import org.apache.cassandra.io.compress.LZ4Compressor;
@@ -33,6 +36,8 @@ import org.apache.cassandra.io.util.DataOutputBuffer;
 import static junit.framework.Assert.assertEquals;
 import static junit.framework.Assert.assertNotSame;
 import static junit.framework.Assert.fail;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
 
 public class HintsDescriptorTest
 {
@@ -100,21 +105,41 @@ public class HintsDescriptorTest
 ImmutableMap parameters = ImmutableMap.of();
 HintsDescriptor expected = new HintsDescriptor(hostId, version, 
timestamp, parameters);
 
-File directory = Files.createTempDir();
+Path directory = Files.createTempDirectory("hints");
 try
 {
-try (HintsW

[cassandra] branch cassandra-3.11 updated (22a0927 -> 228e33d)

2021-01-15 Thread marcuse
This is an automated email from the ASF dual-hosted git repository.

marcuse pushed a change to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from 22a0927  Merge branch 'cassandra-3.0' into cassandra-3.11
 new f02e535  Improve empty/corrupt hint file handling on startup
 new 228e33d  Merge branch 'cassandra-3.0' into cassandra-3.11

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|  1 +
 .../apache/cassandra/hints/HintsDescriptor.java| 28 -
 .../cassandra/hints/HintsDescriptorTest.java   | 35 ++
 3 files changed, 58 insertions(+), 6 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org