[jira] [Created] (CASSANDRA-15874) Bootstrap completes Successfully without streaming all the data

2020-06-12 Thread Jai Bheemsen Rao Dhanwada (Jira)
Jai Bheemsen Rao Dhanwada created CASSANDRA-15874:
-

 Summary: Bootstrap completes Successfully without streaming all 
the data
 Key: CASSANDRA-15874
 URL: https://issues.apache.org/jira/browse/CASSANDRA-15874
 Project: Cassandra
  Issue Type: Bug
  Components: Consistency/Bootstrap and Decommission
Reporter: Jai Bheemsen Rao Dhanwada


I am seeing a strange issue where, adding a new node with auto_bootstrap: true 
is not streaming all the data before it joins the cluster. Don't see any 
information in the logs about bootstrap failures.

Here is the sequence of logs

 
{code:java}
INFO [main] 2020-06-12 01:41:49,642 StorageService.java:1446 - JOINING: schema 
complete, ready to bootstrap
INFO [main] 2020-06-12 01:41:49,642 StorageService.java:1446 - JOINING: waiting 
for pending range calculation
INFO [main] 2020-06-12 01:41:49,643 StorageService.java:1446 - JOINING: 
calculation complete, ready to bootstrap
INFO [main] 2020-06-12 01:41:49,643 StorageService.java:1446 - JOINING: getting 
bootstrap token
INFO [main] 2020-06-12 01:42:19,656 StorageService.java:1446 - JOINING: 
Starting to bootstrap...
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for 
cfId . If a table was just created, this is likely due to the schema not 
being fully propagated. Please wait for schema agreement on table creation.
INFO [StreamReceiveTask:1] 2020-06-12 02:29:51,892 StreamResultFuture.java:219 
- [Stream #f4224f444-a55d-154a-23e3-867899486f5f] All sessions completed INFO 
[StreamReceiveTask:1] 2020-06-12 02:29:51,892 StorageService.java:1505 - 
Bootstrap completed! for the tokens
{code}
Cassandra Version: 3.11.3

I am not able to reproduce this issue all the time, but it happened couple of 
times. Is there any  race condition/corner case, which could cause this issue?

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Commented] (CASSANDRA-15794) Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact tables in downgraded C* (3.11.4) introduces non-existent columns

2020-06-12 Thread Zhuqi Jin (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134575#comment-17134575
 ] 

Zhuqi Jin commented on CASSANDRA-15794:
---

Hi, [~ifesdjeen].

I created patches for 3.0 and 3.11, according to the third method we discussed 
earlier.

[^CASSANDRA-15794-branch-3.0.patch]

I've attached the patches. Would you mind reviewing them? 

> Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact 
> tables in downgraded C* (3.11.4) introduces non-existent columns
> -
>
> Key: CASSANDRA-15794
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15794
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Zhuqi Jin
>Priority: Normal
> Attachments: CASSANDRA-15794-branch-3.0.patch, 
> CASSANDRA-15794-branch-3.11.patch
>
>
> We tried to test upgrading a 3.11.4 C* cluster to 4.x and run into the 
> following problems. 
>  * We started a single 3.11.4 C* node. 
>  * We ran cassandra-stress like this
> {code:java}
> ./cassandra-stress write n = 30 -rate threads = 10 -node  172.17.0.2 {code}
>  * We stopped this node, and started a C* node running C* compiled from trunk 
> (git commit: e394dc0bb32f612a476269010930c617dd1ed3cb)
>  * New C* failed to start with the following error message
> {code:java}
> ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 - Error while 
> loading schema: ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 
> - Error while loading schema: java.lang.IllegalArgumentException: Compact 
> Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... 
> DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to 
> migrate off Compact Storage. at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965)
>  at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924)
>  at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883)
>  at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874)
>  at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862)
>  at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at 
> org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) 
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653)
>  at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770)Exception
>  (java.lang.IllegalArgumentException) encountered during startup: Compact 
> Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... 
> DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to 
> migrate off Compact Storage.ERROR [main] 2020-05-07 00:58:18,520 
> CassandraDaemon.java:792 - Exception encountered during 
> startupjava.lang.IllegalArgumentException: Compact Tables are not allowed in 
> Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE` 
> command supplied in 3.x/3.11 Cassandra in order to migrate off Compact 
> Storage. at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965)
>  at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924)
>  at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883)
>  at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874)
>  at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862)
>  at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at 
> org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) 
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653)
>  at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770){code}
>  * We stopped the trunk version C* and started the 3.11.4 version C*. 
>  * 3.11.4 C* failed to start with the following error messages: 
> {code:java}
> Exception (java.lang.IllegalStateException) encountered during startup: 
> Unknown commitlog version 7Exception (java.lang.IllegalStateException) 
> encountered during startup: Unknown commitlog version 7ERROR [main] 
> 2020-05-07 01:13:26,798 CassandraDaemon.java:749 - Exception encountered 
> during startupjava.lang.IllegalStateException: Unknown commitlog version 7 at 
> org.apache.cassandra.db.commitlog.CommitLogDescriptor.getMessagingVersion(CommitLogDescriptor.java:227)
>  ~[main/:na] at 
> 

[jira] [Updated] (CASSANDRA-15794) Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact tables in downgraded C* (3.11.4) introduces non-existent columns

2020-06-12 Thread Zhuqi Jin (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhuqi Jin updated CASSANDRA-15794:
--
Attachment: CASSANDRA-15794-branch-3.11.patch
CASSANDRA-15794-branch-3.0.patch

> Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact 
> tables in downgraded C* (3.11.4) introduces non-existent columns
> -
>
> Key: CASSANDRA-15794
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15794
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Zhuqi Jin
>Priority: Normal
> Attachments: CASSANDRA-15794-branch-3.0.patch, 
> CASSANDRA-15794-branch-3.11.patch
>
>
> We tried to test upgrading a 3.11.4 C* cluster to 4.x and run into the 
> following problems. 
>  * We started a single 3.11.4 C* node. 
>  * We ran cassandra-stress like this
> {code:java}
> ./cassandra-stress write n = 30 -rate threads = 10 -node  172.17.0.2 {code}
>  * We stopped this node, and started a C* node running C* compiled from trunk 
> (git commit: e394dc0bb32f612a476269010930c617dd1ed3cb)
>  * New C* failed to start with the following error message
> {code:java}
> ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 - Error while 
> loading schema: ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 
> - Error while loading schema: java.lang.IllegalArgumentException: Compact 
> Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... 
> DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to 
> migrate off Compact Storage. at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965)
>  at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924)
>  at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883)
>  at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874)
>  at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862)
>  at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at 
> org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) 
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653)
>  at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770)Exception
>  (java.lang.IllegalArgumentException) encountered during startup: Compact 
> Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... 
> DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to 
> migrate off Compact Storage.ERROR [main] 2020-05-07 00:58:18,520 
> CassandraDaemon.java:792 - Exception encountered during 
> startupjava.lang.IllegalArgumentException: Compact Tables are not allowed in 
> Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE` 
> command supplied in 3.x/3.11 Cassandra in order to migrate off Compact 
> Storage. at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965)
>  at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924)
>  at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883)
>  at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874)
>  at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862)
>  at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at 
> org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) 
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653)
>  at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770){code}
>  * We stopped the trunk version C* and started the 3.11.4 version C*. 
>  * 3.11.4 C* failed to start with the following error messages: 
> {code:java}
> Exception (java.lang.IllegalStateException) encountered during startup: 
> Unknown commitlog version 7Exception (java.lang.IllegalStateException) 
> encountered during startup: Unknown commitlog version 7ERROR [main] 
> 2020-05-07 01:13:26,798 CassandraDaemon.java:749 - Exception encountered 
> during startupjava.lang.IllegalStateException: Unknown commitlog version 7 at 
> org.apache.cassandra.db.commitlog.CommitLogDescriptor.getMessagingVersion(CommitLogDescriptor.java:227)
>  ~[main/:na] at 
> org.apache.cassandra.db.commitlog.CommitLogReader.shouldSkipSegmentId(CommitLogReader.java:276)
>  ~[main/:na] at 
> 

[jira] [Commented] (CASSANDRA-14888) Several mbeans are not unregistered when dropping a keyspace and table

2020-06-12 Thread Caleb Rackliffe (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134568#comment-17134568
 ] 

Caleb Rackliffe commented on CASSANDRA-14888:
-

[~stillalex] The unit and in-JVM dtests look pretty good so far. I'm sorting 
out some hardware issues w/ the dtests, but I'll let you know when those 
resolve.

> Several mbeans are not unregistered when dropping a keyspace and table
> --
>
> Key: CASSANDRA-14888
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14888
> Project: Cassandra
>  Issue Type: Bug
>  Components: Observability/Metrics
>Reporter: Ariel Weisberg
>Assignee: Alex Deparvu
>Priority: Urgent
>  Labels: patch-available
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
> Attachments: CASSANDRA-14888.patch
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> CasCommit, CasPrepare, CasPropose, ReadRepairRequests, 
> ShortReadProtectionRequests, AntiCompactionTime, BytesValidated, 
> PartitionsValidated, RepairPrepareTime, RepairSyncTime, 
> RepairedDataInconsistencies, ViewLockAcquireTime, ViewReadTime, 
> WriteFailedIdealCL
> Basically for 3 years people haven't known what they are doing because the 
> entire thing is kind of obscure. Fix it and also add a dtest that detects if 
> any mbeans are left behind after dropping a table and keyspace.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Commented] (CASSANDRA-15873) Update Netty 4.0.44 -> 4.1.50 (fix security/performance issues)

2020-06-12 Thread Matt Davis (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-15873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134472#comment-17134472
 ] 

Matt Davis commented on CASSANDRA-15873:


Works for me, thanks!

> Update Netty 4.0.44 -> 4.1.50 (fix security/performance issues)
> ---
>
> Key: CASSANDRA-15873
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15873
> Project: Cassandra
>  Issue Type: Task
>  Components: Dependencies
>Reporter: Matt Davis
>Priority: Normal
> Attachments: dependency-check-report.html
>
>
> See https://issues.apache.org/jira/browse/CASSANDRA-15868 for the same issue 
> on 4.0 / trunk. Attached is an OWASP dependency report for Netty 4.0.44, 
> which identifies 3 of the same vulnerabilities as above.
>  
> Additionally, 4.1.50 contains aarch64 native libraries which can improve 
> performance on ARM processors.
>  
> (If the preference is to handle PRs for both versions/branches in a single 
> issue, feel free to close this as a duplicate.)
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-15873) Update Netty 4.0.44 -> 4.1.50 (fix security/performance issues)

2020-06-12 Thread Brandon Williams (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams updated CASSANDRA-15873:
-
Resolution: Duplicate
Status: Resolved  (was: Triage Needed)

Matt,

I updated the fixver for CASSANDRA-15868 to include 3.11.  Let's handle 
everything there.

> Update Netty 4.0.44 -> 4.1.50 (fix security/performance issues)
> ---
>
> Key: CASSANDRA-15873
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15873
> Project: Cassandra
>  Issue Type: Task
>  Components: Dependencies
>Reporter: Matt Davis
>Priority: Normal
> Attachments: dependency-check-report.html
>
>
> See https://issues.apache.org/jira/browse/CASSANDRA-15868 for the same issue 
> on 4.0 / trunk. Attached is an OWASP dependency report for Netty 4.0.44, 
> which identifies 3 of the same vulnerabilities as above.
>  
> Additionally, 4.1.50 contains aarch64 native libraries which can improve 
> performance on ARM processors.
>  
> (If the preference is to handle PRs for both versions/branches in a single 
> issue, feel free to close this as a duplicate.)
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-15868) Update Netty version to 4.1.50 because there are security issues in 4.1.37

2020-06-12 Thread Brandon Williams (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams updated CASSANDRA-15868:
-
Fix Version/s: 4.0-beta
   3.11.7

> Update Netty version to 4.1.50 because there are security issues in 4.1.37
> --
>
> Key: CASSANDRA-15868
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15868
> Project: Cassandra
>  Issue Type: Task
>  Components: Dependencies
>Reporter: Stefan Miklosovic
>Priority: Normal
> Fix For: 3.11.7, 4.0-beta
>
> Attachments: dependency-check-report.html
>
>
> Please see attached HTML report from OWASP dependency check.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Comment Edited] (CASSANDRA-15867) Update Jackson version to 2.9.10.1 because there are security issues in 2.9.5

2020-06-12 Thread Brandon Williams (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-15867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134464#comment-17134464
 ] 

Brandon Williams edited comment on CASSANDRA-15867 at 6/12/20, 6:50 PM:


I would say if we fix it in one branch, but another is also vulnerable for the 
same reason, we should fix it there too.

bq. This holds for more dependencies, what is the general approach here?

I would take it on a case-by-case basis.  I looked into the Jackson 
vulnerability and it does seem to be exploitable for us (though I don't know 
why users would DoS their database on purpose, certainly accidents can happen.)


was (Author: brandon.williams):
I would say if we fix it in one branch, but another is also vulnerable for the 
same reason, we should fix it there too.

> This holds for more dependencies, what is the general approach here?

I would take it on a case-by-case basis.  I looked into the Jackson 
vulnerability and it does seem to be exploitable for us (though I don't know 
why users would DoS their database on purpose, certainly accidents can happen.)

> Update Jackson version to 2.9.10.1 because there are security issues in 2.9.5
> -
>
> Key: CASSANDRA-15867
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15867
> Project: Cassandra
>  Issue Type: Task
>  Components: Dependencies
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0-alpha5
>
> Attachments: dependency-check-report.html
>
>
> Please see attached HTML report from OWASP dependency check for current 
> 4.0-alpha5 trunk branch.
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Commented] (CASSANDRA-15867) Update Jackson version to 2.9.10.1 because there are security issues in 2.9.5

2020-06-12 Thread Brandon Williams (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-15867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134464#comment-17134464
 ] 

Brandon Williams commented on CASSANDRA-15867:
--

I would say if we fix it in one branch, but another is also vulnerable for the 
same reason, we should fix it there too.

> This holds for more dependencies, what is the general approach here?

I would take it on a case-by-case basis.  I looked into the Jackson 
vulnerability and it does seem to be exploitable for us (though I don't know 
why users would DoS their database on purpose, certainly accidents can happen.)

> Update Jackson version to 2.9.10.1 because there are security issues in 2.9.5
> -
>
> Key: CASSANDRA-15867
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15867
> Project: Cassandra
>  Issue Type: Task
>  Components: Dependencies
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0-alpha5
>
> Attachments: dependency-check-report.html
>
>
> Please see attached HTML report from OWASP dependency check for current 
> 4.0-alpha5 trunk branch.
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Commented] (CASSANDRA-15867) Update Jackson version to 2.9.10.1 because there are security issues in 2.9.5

2020-06-12 Thread Stefan Miklosovic (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-15867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134461#comment-17134461
 ] 

Stefan Miklosovic commented on CASSANDRA-15867:
---

[~brandon.williams] I tried to apply it on 3.11 but there are incompatibilies 
on a source-code levels (classes not there etc).

I do not want to make any changes to the codebase of 3.11 unnecessarilly, what 
is your opinion here? Should I invest time to bump it while it is not so 
trivial? This holds for more dependencies, what is the general approach here?

> Update Jackson version to 2.9.10.1 because there are security issues in 2.9.5
> -
>
> Key: CASSANDRA-15867
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15867
> Project: Cassandra
>  Issue Type: Task
>  Components: Dependencies
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0-alpha5
>
> Attachments: dependency-check-report.html
>
>
> Please see attached HTML report from OWASP dependency check for current 
> 4.0-alpha5 trunk branch.
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Comment Edited] (CASSANDRA-15867) Update Jackson version to 2.9.10.1 because there are security issues in 2.9.5

2020-06-12 Thread Stefan Miklosovic (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-15867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134461#comment-17134461
 ] 

Stefan Miklosovic edited comment on CASSANDRA-15867 at 6/12/20, 6:43 PM:
-

[~brandon.williams] I tried to apply it on 3.11 but there are incompatibilies 
on source-code level (classes not there etc).

I do not want to make any changes to the codebase of 3.11 unnecessarilly, what 
is your opinion here? Should I invest time to bump it while it is not so 
trivial? This holds for more dependencies, what is the general approach here?


was (Author: stefan.miklosovic):
[~brandon.williams] I tried to apply it on 3.11 but there are incompatibilies 
on a source-code levels (classes not there etc).

I do not want to make any changes to the codebase of 3.11 unnecessarilly, what 
is your opinion here? Should I invest time to bump it while it is not so 
trivial? This holds for more dependencies, what is the general approach here?

> Update Jackson version to 2.9.10.1 because there are security issues in 2.9.5
> -
>
> Key: CASSANDRA-15867
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15867
> Project: Cassandra
>  Issue Type: Task
>  Components: Dependencies
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0-alpha5
>
> Attachments: dependency-check-report.html
>
>
> Please see attached HTML report from OWASP dependency check for current 
> 4.0-alpha5 trunk branch.
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Comment Edited] (CASSANDRA-15868) Update Netty version to 4.1.50 because there are security issues in 4.1.37

2020-06-12 Thread Stefan Miklosovic (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-15868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17130112#comment-17130112
 ] 

Stefan Miklosovic edited comment on CASSANDRA-15868 at 6/12/20, 6:40 PM:
-

[https://github.com/apache/cassandra/pull/631/files]


was (Author: stefan.miklosovic):
[https://github.com/apache/cassandra/compare/trunk...smiklosovic:CASSANDRA-15868?expand=1]

> Update Netty version to 4.1.50 because there are security issues in 4.1.37
> --
>
> Key: CASSANDRA-15868
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15868
> Project: Cassandra
>  Issue Type: Task
>  Components: Dependencies
>Reporter: Stefan Miklosovic
>Priority: Normal
> Attachments: dependency-check-report.html
>
>
> Please see attached HTML report from OWASP dependency check.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Commented] (CASSANDRA-15868) Update Netty version to 4.1.50 because there are security issues in 4.1.37

2020-06-12 Thread Matt Davis (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-15868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134454#comment-17134454
 ] 

Matt Davis commented on CASSANDRA-15868:


Note also that 4.1.50 contains aarch64 native libraries for improved 
performance on ARM processors. See 
https://issues.apache.org/jira/browse/CASSANDRA-15873 for the same issue on 
Cassandra 3.11 (Netty 4.0.44 -> 4.1.50).

> Update Netty version to 4.1.50 because there are security issues in 4.1.37
> --
>
> Key: CASSANDRA-15868
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15868
> Project: Cassandra
>  Issue Type: Task
>  Components: Dependencies
>Reporter: Stefan Miklosovic
>Priority: Normal
> Attachments: dependency-check-report.html
>
>
> Please see attached HTML report from OWASP dependency check.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-15873) Update Netty 4.0.44 -> 4.1.50 (fix security/performance issues)

2020-06-12 Thread Matt Davis (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Davis updated CASSANDRA-15873:
---
Description: 
See https://issues.apache.org/jira/browse/CASSANDRA-15868 for the same issue on 
4.0 / trunk. Attached is an OWASP dependency report for Netty 4.0.44, which 
identifies 3 of the same vulnerabilities as above.

 

Additionally, 4.1.50 contains aarch64 native libraries which can improve 
performance on ARM processors.

 

(If the preference is to handle PRs for both versions/branches in a single 
issue, feel free to close this as a duplicate.)

 

 

  was:
See https://issues.apache.org/jira/browse/CASSANDRA-15868 for the same issue on 
4.0 / trunk. Attached is an OWASP dependency report for Netty 4.0.44, which 
identifies 3 of the same vulnerabilities as the above issue.

 

Additionally, 4.1.50 contains aarch64 native libraries which can improve 
performance on ARM processors.

 

(If the preference is to handle PRs for both versions/branches in a single 
issue, feel free to close this as a duplicate.)

 

 


> Update Netty 4.0.44 -> 4.1.50 (fix security/performance issues)
> ---
>
> Key: CASSANDRA-15873
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15873
> Project: Cassandra
>  Issue Type: Task
>  Components: Dependencies
>Reporter: Matt Davis
>Priority: Normal
> Attachments: dependency-check-report.html
>
>
> See https://issues.apache.org/jira/browse/CASSANDRA-15868 for the same issue 
> on 4.0 / trunk. Attached is an OWASP dependency report for Netty 4.0.44, 
> which identifies 3 of the same vulnerabilities as above.
>  
> Additionally, 4.1.50 contains aarch64 native libraries which can improve 
> performance on ARM processors.
>  
> (If the preference is to handle PRs for both versions/branches in a single 
> issue, feel free to close this as a duplicate.)
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Created] (CASSANDRA-15873) Update Netty 4.0.44 -> 4.1.50 (fix security/performance issues)

2020-06-12 Thread Matt Davis (Jira)
Matt Davis created CASSANDRA-15873:
--

 Summary: Update Netty 4.0.44 -> 4.1.50 (fix security/performance 
issues)
 Key: CASSANDRA-15873
 URL: https://issues.apache.org/jira/browse/CASSANDRA-15873
 Project: Cassandra
  Issue Type: Task
  Components: Dependencies
Reporter: Matt Davis
 Attachments: dependency-check-report.html

See https://issues.apache.org/jira/browse/CASSANDRA-15868 for the same issue on 
4.0 / trunk. Attached is an OWASP dependency report for Netty 4.0.44, which 
identifies 3 of the same vulnerabilities as the above issue.

 

Additionally, 4.1.50 contains aarch64 native libraries which can improve 
performance on ARM processors.

 

(If the preference is to handle PRs for both versions/branches in a single 
issue, feel free to close this as a duplicate.)

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-15299) CASSANDRA-13304 follow-up: improve checksumming and compression in protocol v5-beta

2020-06-12 Thread ZhaoYang (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ZhaoYang updated CASSANDRA-15299:
-
Reviewers: ZhaoYang

> CASSANDRA-13304 follow-up: improve checksumming and compression in protocol 
> v5-beta
> ---
>
> Key: CASSANDRA-15299
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15299
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Messaging/Client
>Reporter: Aleksey Yeschenko
>Assignee: Sam Tunnicliffe
>Priority: Normal
>  Labels: protocolv5
> Fix For: 4.0-alpha
>
>
> CASSANDRA-13304 made an important improvement to our native protocol: it 
> introduced checksumming/CRC32 to request and response bodies. It’s an 
> important step forward, but it doesn’t cover the entire stream. In 
> particular, the message header is not covered by a checksum or a crc, which 
> poses a correctness issue if, for example, {{streamId}} gets corrupted.
> Additionally, we aren’t quite using CRC32 correctly, in two ways:
> 1. We are calculating the CRC32 of the *decompressed* value instead of 
> computing the CRC32 on the bytes written on the wire - losing the properties 
> of the CRC32. In some cases, due to this sequencing, attempting to decompress 
> a corrupt stream can cause a segfault by LZ4.
> 2. When using CRC32, the CRC32 value is written in the incorrect byte order, 
> also losing some of the protections.
> See https://users.ece.cmu.edu/~koopman/pubs/KoopmanCRCWebinar9May2012.pdf for 
> explanation for the two points above.
> Separately, there are some long-standing issues with the protocol - since 
> *way* before CASSANDRA-13304. Importantly, both checksumming and compression 
> operate on individual message bodies rather than frames of multiple complete 
> messages. In reality, this has several important additional downsides. To 
> name a couple:
> # For compression, we are getting poor compression ratios for smaller 
> messages - when operating on tiny sequences of bytes. In reality, for most 
> small requests and responses we are discarding the compressed value as it’d 
> be smaller than the uncompressed one - incurring both redundant allocations 
> and compressions.
> # For checksumming and CRC32 we pay a high overhead price for small messages. 
> 4 bytes extra is *a lot* for an empty write response, for example.
> To address the correctness issue of {{streamId}} not being covered by the 
> checksum/CRC32 and the inefficiency in compression and checksumming/CRC32, we 
> should switch to a framing protocol with multiple messages in a single frame.
> I suggest we reuse the framing protocol recently implemented for internode 
> messaging in CASSANDRA-15066 to the extent that its logic can be borrowed, 
> and that we do it before native protocol v5 graduates from beta. See 
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderCrc.java
>  and 
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderLZ4.java.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Commented] (CASSANDRA-15752) Range read concurrency factor didn't consider range merger

2020-06-12 Thread Caleb Rackliffe (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-15752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134358#comment-17134358
 ] 

Caleb Rackliffe commented on CASSANDRA-15752:
-

+1

> Range read concurrency factor didn't consider range merger
> --
>
> Key: CASSANDRA-15752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15752
> Project: Cassandra
>  Issue Type: Bug
>  Components: Legacy/Coordination
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
>
> During range read, coordinator computes concurrency factor which is the 
> number of vnode ranges to contact in parallel for the next batch.
> But in {{RangeCommandIterator}}, vnode ranges are merged by {{RangeMerger}} 
> if vnode ranges share enough replicas to satisfy consistency level. eg. vnode 
> range [a,b) has replica n1,n2,n3 and vnode range [b,c) has replica n2,n3,n4, 
> so they can be merged as range [a,c) with replica n2, n3 for Quorum.
> Currently it counts number of merged ranges towards concurrency factor. 
> Coordinator may fetch more ranges than needed.
> 
> Another issue is that when executing range read on table with very small 
> amount of data, concurrency factor can be bumped to {{size of total vnode 
> ranges}}, eg. 10k, depending on the num of vnodes and cluster size. As a 
> result, coordinator will send large number of concurrent range requests, 
> potentially slowing down the cluster.. We should cap the max concurrency 
> factor..



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-14819) Certain table metrics not released when table is dropped.

2020-06-12 Thread Caleb Rackliffe (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Caleb Rackliffe updated CASSANDRA-14819:

Resolution: Duplicate
Status: Resolved  (was: Open)

CASSANDRA-14888 includes fixes for these metric releases.

> Certain table metrics not released when table is dropped.
> -
>
> Key: CASSANDRA-14819
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14819
> Project: Cassandra
>  Issue Type: Bug
>  Components: Observability/Metrics
>Reporter: Adam Zegelin
>Priority: Low
> Attachments: release-metrics.patch
>
>
> The following metrics are not released when a table is dropped:
>  * {{CasCommitLatency}}
>  * {{CasCommitTotalLatency}}
>  * {{CasPrepareLatency}}
>  * {{CasPrepareTotalLatency}}
>  * {{CasProposeLatency}}
>  * {{CasProposeTotalLatency}}
>  * {{ReadRepairRequests}}
>  * {{ShortReadProtectionRequests}}
>  
> This can be verified by inspecting the list of exported MBeans after creating 
> a table then dropping it.
> The result is that if a table with the same name is re-created then the above 
> metrics wont be updated - internally a 
> {{javax.management.InstanceAlreadyExistsException}} is thrown by the 
> {{MBeanServer}} when the new table's metric is registered, but this exception 
> is silently ignored by 
> {{org.apache.cassandra.metrics.CassandraMetricsRegistry#registerMBean}}
> The above metrics were added in 2014/2015, so this issue probably affects all 
> versions released since then.
>  
> Attached a patch that releases/removes the metrics when the table is dropped.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-14888) Several mbeans are not unregistered when dropping a keyspace and table

2020-06-12 Thread Caleb Rackliffe (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Caleb Rackliffe updated CASSANDRA-14888:

Test and Documentation Plan: 
Review and tests
CircleCI: 
https://app.circleci.com/pipelines/github/maedhroz/cassandra?branch=14888-maedhroz

  was:
Review and tests
CircleCI: 
https://app.circleci.com/pipelines/github/maedhroz/cassandra/5/workflows/357e99c0-fa27-4027-8b01-3909eac71e4c


> Several mbeans are not unregistered when dropping a keyspace and table
> --
>
> Key: CASSANDRA-14888
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14888
> Project: Cassandra
>  Issue Type: Bug
>  Components: Observability/Metrics
>Reporter: Ariel Weisberg
>Assignee: Alex Deparvu
>Priority: Urgent
>  Labels: patch-available
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
> Attachments: CASSANDRA-14888.patch
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> CasCommit, CasPrepare, CasPropose, ReadRepairRequests, 
> ShortReadProtectionRequests, AntiCompactionTime, BytesValidated, 
> PartitionsValidated, RepairPrepareTime, RepairSyncTime, 
> RepairedDataInconsistencies, ViewLockAcquireTime, ViewReadTime, 
> WriteFailedIdealCL
> Basically for 3 years people haven't known what they are doing because the 
> entire thing is kind of obscure. Fix it and also add a dtest that detects if 
> any mbeans are left behind after dropping a table and keyspace.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Commented] (CASSANDRA-14888) Several mbeans are not unregistered when dropping a keyspace and table

2020-06-12 Thread Alex Deparvu (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134312#comment-17134312
 ] 

Alex Deparvu commented on CASSANDRA-14888:
--

thanks [~maedhroz] let me know how the CI results look like.
just in case it got missed in the wall of updates, this PR also includes 
CASSANDRA-14819.

> Several mbeans are not unregistered when dropping a keyspace and table
> --
>
> Key: CASSANDRA-14888
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14888
> Project: Cassandra
>  Issue Type: Bug
>  Components: Observability/Metrics
>Reporter: Ariel Weisberg
>Assignee: Alex Deparvu
>Priority: Urgent
>  Labels: patch-available
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
> Attachments: CASSANDRA-14888.patch
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> CasCommit, CasPrepare, CasPropose, ReadRepairRequests, 
> ShortReadProtectionRequests, AntiCompactionTime, BytesValidated, 
> PartitionsValidated, RepairPrepareTime, RepairSyncTime, 
> RepairedDataInconsistencies, ViewLockAcquireTime, ViewReadTime, 
> WriteFailedIdealCL
> Basically for 3 years people haven't known what they are doing because the 
> entire thing is kind of obscure. Fix it and also add a dtest that detects if 
> any mbeans are left behind after dropping a table and keyspace.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-15861) Mutating sstable STATS metadata may race with entire-sstable-streaming(ZCS) causing checksum validation failure

2020-06-12 Thread ZhaoYang (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ZhaoYang updated CASSANDRA-15861:
-
Description: 
Flaky dtest: [test_dead_sync_initiator - 
repair_tests.repair_test.TestRepair|https://ci-cassandra.apache.org/view/all/job/Cassandra-devbranch-dtest/143/testReport/junit/dtest.repair_tests.repair_test/TestRepair/test_dead_sync_initiator/]

In the above test, it executes "nodetool repair" on node1 and kills node2 
during repair. At the end, node3 reports checksum validation failure on sstable 
transferred from node1.
{code:java|title=what happened}
1. When repair started on node1, it performs anti-compaction which modifies 
sstable's repairAt to 0 and pending repair id to session-id.
2. Then node1 creates {{ComponentManifest}} which contains file lengths to be 
transferred to node3.
3. Before node1 actually sends the files to node3, node2 is killed and node1 
starts to broadcast repair-failure-message to all participants in 
{{CoordinatorSession#fail}}
4. Node1 receives its own repair-failure-message and fails its local repair 
sessions at {{LocalSessions#failSession}} which triggers async background 
compaction.
5. Node1's background compaction will mutate sstable's repairAt to 0 and 
pending repair id to null via  
{{PendingRepairManager#getNextRepairFinishedTask}}, as there is no more 
in-progress repair.
6. Node1 actually sends the sstable to node3 where the sstable's STATS 
component size is different from the original size recorded in the manifest.
7. At the end, node3 reports checksum validation failure when it tries to 
mutate sstable level and "isTransient" attribute in 
{{CassandraEntireSSTableStreamReader#read}}.
{code}

I believe similar race may happen with level compaction where it may directly 
mutate a sstable's level if it doesn't overlap with sstables at next level. 
(Note: this isn't a problem in legacy streaming as STATS file length didn't 
matter.)

Ideally it will be great to make sstable STATS metadata immutable, just like 
other sstable components, so we don't have to worry this special case. 

I can think of two ways:
# Change {{RepairFinishedCompactionTask}} and {{SingleSSTableLCSTask}} to 
create hard link on the compacting sstable components with a new descriptor, 
except STATS files which will be copied entirely. Then mutation will be applied 
on the new STATS file. At the end, old sstable will be released. This ensures 
all sstable components are immutable and shouldn't make these special 
compaction tasks slower.
# Hacky approach: load the small STATS file into memory when initializing 
{{CassandraOutgoingFile}} instead of relying on mutable on-disk STATS file.

  was:
Flaky dtest: [test_dead_sync_initiator - 
repair_tests.repair_test.TestRepair|https://ci-cassandra.apache.org/view/all/job/Cassandra-devbranch-dtest/143/testReport/junit/dtest.repair_tests.repair_test/TestRepair/test_dead_sync_initiator/]

In the above test, it executes "nodetool repair" on node1 and kills node2 
during repair. At the end, node3 reports checksum validation failure on sstable 
transferred from node1.
{code:java|title=what happened}
1. When repair started on node1, it performs anti-compaction which modifies 
sstable's repairAt to 0 and pending repair id to session-id.
2. Then node1 creates {{ComponentManifest}} which contains file lengths to be 
transferred to node3.
3. Before node1 actually sends the files to node3, node2 is killed and node1 
starts to broadcast repair-failure-message to all participants in 
{{CoordinatorSession#fail}}
4. Node1 receives its own repair-failure-message and fails its local repair 
sessions at {{LocalSessions#failSession}} which triggers async background 
compaction.
5. Node1's background compaction will mutate sstable's repairAt to 0 and 
pending repair id to null via  
{{PendingRepairManager#getNextRepairFinishedTask}}, as there is no more 
in-progress repair.
6. Node1 actually sends the sstable to node3 where the sstable's STATS 
component size is different from the original size recorded in the manifest.
7. At the end, node3 reports checksum validation failure when it tries to 
mutate sstable level and "isTransient" attribute in 
{{CassandraEntireSSTableStreamReader#read}}.
{code}

I believe similar race may happen with level compaction where it may directly 
mutate a sstable's level if it doesn't overlap with sstables at next level. 
(Note: this isn't a problem in legacy streaming as STATS file length didn't 
matter.)

Ideally it will be great to make sstable STATS metadata immutable, just like 
other sstable components, so we don't have to worry this special case. 

I can think of two ways:
# Change {{RepairFinishedCompactionTask}} and {{SingleSSTableLCSTask}} to 
create hard link on the compacting sstable components with a new descriptor, 
except STATS files which will be copied entirely. Then mutation will be applied 
on the new STATS file. 

[jira] [Updated] (CASSANDRA-15861) Mutating sstable STATS metadata may race with entire-sstable-streaming(ZCS) causing checksum validation failure

2020-06-12 Thread ZhaoYang (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ZhaoYang updated CASSANDRA-15861:
-
Description: 
Flaky dtest: [test_dead_sync_initiator - 
repair_tests.repair_test.TestRepair|https://ci-cassandra.apache.org/view/all/job/Cassandra-devbranch-dtest/143/testReport/junit/dtest.repair_tests.repair_test/TestRepair/test_dead_sync_initiator/]

In the above test, it executes "nodetool repair" on node1 and kills node2 
during repair. At the end, node3 reports checksum validation failure on sstable 
transferred from node1.
{code:java|title=what happened}
1. When repair started on node1, it performs anti-compaction which modifies 
sstable's repairAt to 0 and pending repair id to session-id.
2. Then node1 creates {{ComponentManifest}} which contains file lengths to be 
transferred to node3.
3. Before node1 actually sends the files to node3, node2 is killed and node1 
starts to broadcast repair-failure-message to all participants in 
{{CoordinatorSession#fail}}
4. Node1 receives its own repair-failure-message and fails its local repair 
sessions at {{LocalSessions#failSession}} which triggers async background 
compaction.
5. Node1's background compaction will mutate sstable's repairAt to 0 and 
pending repair id to null via  
{{PendingRepairManager#getNextRepairFinishedTask}}, as there is no more 
in-progress repair.
6. Node1 actually sends the sstable to node3 where the sstable's STATS 
component size is different from the original size recorded in the manifest.
7. At the end, node3 reports checksum validation failure when it tries to 
mutate sstable level and "isTransient" attribute in 
{{CassandraEntireSSTableStreamReader#read}}.
{code}

I believe similar race may happen with level compaction where it may directly 
mutate a sstable's level if it doesn't overlap with sstables at next level. 
(Note: this isn't a problem in legacy streaming as STATS file length didn't 
matter.)

Ideally it will be great to make sstable STATS metadata immutable, just like 
other sstable components, so we don't have to worry this special case. 

I can think of two ways:
# Change {{RepairFinishedCompactionTask}} and {{SingleSSTableLCSTask}} to 
create hard link on the compacting sstable components with a new descriptor, 
except STATS files which will be copied entirely. Then mutation will be applied 
on the new STATS file. At the end, old sstable will be released. This ensures 
all sstable components are immutable.
# Hacky approach: load the small STATS file into memory when initializing 
{{CassandraOutgoingFile}} instead of relying on mutable on-disk STATS file.

  was:
Flaky dtest: [test_dead_sync_initiator - 
repair_tests.repair_test.TestRepair|https://ci-cassandra.apache.org/view/all/job/Cassandra-devbranch-dtest/143/testReport/junit/dtest.repair_tests.repair_test/TestRepair/test_dead_sync_initiator/]

In the above test, it executes "nodetool repair" on node1 and kills node2 
during repair. At the end, node3 reports checksum validation failure on sstable 
transferred from node1.
{code:java|title=what happened}
1. When repair started on node1, it performs anti-compaction which modifies 
sstable's repairAt to 0 and pending repair id to session-id.
2. Then node1 creates {{ComponentManifest}} which contains file lengths to be 
transferred to node3.
3. Before node1 actually sends the files to node3, node2 is killed and node1 
starts to broadcast repair-failure-message to all participants in 
{{CoordinatorSession#fail}}
4. Node1 receives its own repair-failure-message and fails its local repair 
sessions at {{LocalSessions#failSession}} which triggers async background 
compaction.
5. Node1's background compaction will mutate sstable's repairAt to 0 and 
pending repair id to null via  
{{PendingRepairManager#getNextRepairFinishedTask}}, as there is no more 
in-progress repair.
6. Node1 actually sends the sstable to node3 where the sstable's STATS 
component size is different from the original size recorded in the manifest.
7. At the end, node3 reports checksum validation failure when it tries to 
mutate sstable level and "isTransient" attribute in 
{{CassandraEntireSSTableStreamReader#read}}.
{code}

I believe similar race may happen with level compaction where it may directly 
mutate a sstable's level if it doesn't overlap with sstables at next level. 
(Note: this isn't a problem in legacy streaming as STATS file length didn't 
matter.)

Ideally it will be great to make sstable STATS metadata immutable, just like 
other sstable components, so we don't have to worry this special case. 

I can think of two ways:
1. Change {{RepairFinishedCompactionTask}} and {{SingleSSTableLCSTask}} to 
create hard link on the compacting sstable components with a new descriptor, 
except STATS files which will be copied entirely. Then mutation will be applied 
on the new STATS file. At the end, old sstable will be released. This ensures 

[jira] [Updated] (CASSANDRA-15861) Mutating sstable STATS metadata may race with entire-sstable-streaming(ZCS) causing checksum validation failure

2020-06-12 Thread ZhaoYang (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ZhaoYang updated CASSANDRA-15861:
-
Description: 
Flaky dtest: [test_dead_sync_initiator - 
repair_tests.repair_test.TestRepair|https://ci-cassandra.apache.org/view/all/job/Cassandra-devbranch-dtest/143/testReport/junit/dtest.repair_tests.repair_test/TestRepair/test_dead_sync_initiator/]

In the above test, it executes "nodetool repair" on node1 and kills node2 
during repair. At the end, node3 reports checksum validation failure on sstable 
transferred from node1.
{code:java|title=what happened}
1. When repair started on node1, it performs anti-compaction which modifies 
sstable's repairAt to 0 and pending repair id to session-id.
2. Then node1 creates {{ComponentManifest}} which contains file lengths to be 
transferred to node3.
3. Before node1 actually sends the files to node3, node2 is killed and node1 
starts to broadcast repair-failure-message to all participants in 
{{CoordinatorSession#fail}}
4. Node1 receives its own repair-failure-message and fails its local repair 
sessions at {{LocalSessions#failSession}} which triggers async background 
compaction.
5. Node1's background compaction will mutate sstable's repairAt to 0 and 
pending repair id to null via  
{{PendingRepairManager#getNextRepairFinishedTask}}, as there is no more 
in-progress repair.
6. Node1 actually sends the sstable to node3 where the sstable's STATS 
component size is different from the original size recorded in the manifest.
7. At the end, node3 reports checksum validation failure when it tries to 
mutate sstable level and "isTransient" attribute in 
{{CassandraEntireSSTableStreamReader#read}}.
{code}

I believe similar race may happen with level compaction where it may directly 
mutate a sstable's level if it doesn't overlap with sstables at next level. 
(Note: this isn't a problem in legacy streaming as STATS file length didn't 
matter.)

Ideally it will be great to make sstable STATS metadata immutable, just like 
other sstable components, so we don't have to worry this special case. 

I can think of two ways:
1. Change {{RepairFinishedCompactionTask}} and {{SingleSSTableLCSTask}} to 
create hard link on the compacting sstable components with a new descriptor, 
except STATS files which will be copied entirely. Then mutation will be applied 
on the new STATS file. At the end, old sstable will be released. This ensures 
all sstable components are immutable.
2. Hacky approach: to load the small STATS file into memory when initializing 
{{CassandraOutgoingFile}} instead of relying on mutable on-disk STATS file.

  was:
Flaky dtest: [test_dead_sync_initiator - 
repair_tests.repair_test.TestRepair|https://ci-cassandra.apache.org/view/all/job/Cassandra-devbranch-dtest/143/testReport/junit/dtest.repair_tests.repair_test/TestRepair/test_dead_sync_initiator/]

In the above test, it executes "nodetool repair" on node1 and kills node2 
during repair. At the end, node3 reports checksum validation failure on sstable 
transferred from node1.
{code:java|title=what happened}
1. When repair started on node1, it performs anti-compaction which modifies 
sstable's repairAt to 0 and pending repair id to session-id.
2. Then node1 creates {{ComponentManifest}} which contains file lengths to be 
transferred to node3.
3. Before node1 actually sends the files to node3, node2 is killed and node1 
starts to broadcast repair-failure-message to all participants in 
{{CoordinatorSession#fail}}
4. Node1 receives its own repair-failure-message and fails its local repair 
sessions at {{LocalSessions#failSession}} which triggers async background 
compaction.
5. Node1's background compaction will mutate sstable's repairAt to 0 and 
pending repair id to null via  
{{PendingRepairManager#getNextRepairFinishedTask}}, as there is no more 
in-progress repair.
6. Node1 actually sends the sstable to node3 where the sstable's STATS 
component size is different from the original size recorded in the manifest.
7. At the end, node3 reports checksum validation failure when it tries to 
mutate sstable level and "isTransient" attribute in 
{{CassandraEntireSSTableStreamReader#read}}.
{code}

I believe similar race may happen with level compaction where it may directly 
mutate a sstable's level if it doesn't overlap with sstables at next level. 
(Note: this isn't a problem in legacy streaming as STATS file length didn't 
matter.)

Ideally it will be great to make sstable STATS metadata immutable, just like 
other sstable components, so we don't have to worry this special case. For now, 
I suggest to use a {{StatsMetadata}} snapshot when initializing 
{{CassandraOutgoingFile}} instead of relying on mutable on-disk STATS file.


> Mutating sstable STATS metadata may race with entire-sstable-streaming(ZCS) 
> causing checksum validation failure
> 

[jira] [Updated] (CASSANDRA-14227) Extend maximum expiration date

2020-06-12 Thread Michael Shuler (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-14227:
---
Complexity: Challenging
  Severity: Critical  (was: Normal)

Set this Jira to Severity = Critical and Complexity = Challenging to reflect 
the state of this problem, as time goes on, as well as the nature of the fix.

> Extend maximum expiration date
> --
>
> Key: CASSANDRA-14227
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14227
> Project: Cassandra
>  Issue Type: Bug
>  Components: Legacy/Local Write-Read Paths
>Reporter: Paulo Motta
>Priority: Urgent
>
> The maximum expiration timestamp that can be represented by the storage 
> engine is
> 2038-01-19T03:14:06+00:00 due to the encoding of {{localExpirationTime}} as 
> an int32.
> On CASSANDRA-14092 we added an overflow policy which rejects requests with 
> expiration above the maximum date as a temporary measure, but we should 
> remove this limitation by updating the storage engine to support at least the 
> maximum allowed TTL of 20 years.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Commented] (CASSANDRA-15833) Unresolvable false digest mismatch during upgrade due to CASSANDRA-10657

2020-06-12 Thread Jacek Lewandowski (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-15833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134297#comment-17134297
 ] 

Jacek Lewandowski commented on CASSANDRA-15833:
---

Yup, I'll finish that, just was a bit busy recently

> Unresolvable false digest mismatch during upgrade due to CASSANDRA-10657
> 
>
> Key: CASSANDRA-15833
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15833
> Project: Cassandra
>  Issue Type: Bug
>  Components: Consistency/Repair
>Reporter: Jacek Lewandowski
>Assignee: Jacek Lewandowski
>Priority: Normal
> Fix For: 3.11.x, 4.0-alpha
>
> Attachments: CASSANDRA-15833-3.11.patch, CASSANDRA-15833-4.0.patch
>
>
> CASSANDRA-10657 introduced changes in how the ColumnFilter is interpreted. 
> This results in digest mismatch when querying incomplete set of columns from 
> a table with consistency that requires reaching instances running pre 
> CASSANDRA-10657 from nodes that include CASSANDRA-10657 (it was introduced in 
> Cassandra 3.4). 
> The fix is to bring back the previous behaviour until there are no instances 
> running pre CASSANDRA-10657 version. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Commented] (CASSANDRA-15869) MemoryOutputStream overflow on large bloom filter

2020-06-12 Thread ZhaoYang (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-15869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134296#comment-17134296
 ] 

ZhaoYang commented on CASSANDRA-15869:
--

thank you

> MemoryOutputStream overflow on large bloom filter
> -
>
> Key: CASSANDRA-15869
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15869
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/SSTable
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 4.0-alpha5
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Since CASSANDRA-9067, Cassandra will use {{MemoryOutputStream}} to 
> reconstruct BF Memory without re-ordering bytes, see 
> {{OffsetBitSet#deserialize}}. But {{MemoryOutputStream}} use {{INT}} to track 
> position and will overflow when BF size exceeds 2GB.
> {code:java|title=stacktrace}
> error: Illegal bounds [-2147483648..-2147483584); size: 4588588016
> -- StackTrace --
> java.lang.AssertionError: Illegal bounds [-2147483648..-2147483584); size: 
> 4588588016
>   at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:185)
>   at org.apache.cassandra.io.util.Memory.setBytes(Memory.java:138)
>   at 
> org.apache.cassandra.io.util.MemoryOutputStream.write(MemoryOutputStream.java:45)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Commented] (CASSANDRA-15872) Flaky test: testNoTreesRetainedAfterDifference - org.apache.cassandra.repair.RepairJobTest

2020-06-12 Thread ZhaoYang (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134293#comment-17134293
 ] 

ZhaoYang commented on CASSANDRA-15872:
--

[patch|https://github.com/apache/cassandra/pull/630/files]: the test is flaky 
when repair task executor executes the job before the assertion. the fix is to 
put a latch.

> Flaky test: testNoTreesRetainedAfterDifference - 
> org.apache.cassandra.repair.RepairJobTest
> --
>
> Key: CASSANDRA-15872
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15872
> Project: Cassandra
>  Issue Type: Task
>  Components: Consistency/Repair
>Reporter: ZhaoYang
>Priority: Normal
> Fix For: 4.0-rc
>
>
> link: [https://circleci.com/gh/maedhroz/cassandra/21#tests/containers/95]
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Comment Edited] (CASSANDRA-15872) Flaky test: testNoTreesRetainedAfterDifference - org.apache.cassandra.repair.RepairJobTest

2020-06-12 Thread ZhaoYang (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134293#comment-17134293
 ] 

ZhaoYang edited comment on CASSANDRA-15872 at 6/12/20, 2:58 PM:


[patch|https://github.com/apache/cassandra/pull/630/files]: the test is flaky 
when repair task executor executes the job before the assertion. the fix is to 
put a latch.  cc [~maedhroz]


was (Author: jasonstack):
[patch|https://github.com/apache/cassandra/pull/630/files]: the test is flaky 
when repair task executor executes the job before the assertion. the fix is to 
put a latch.

> Flaky test: testNoTreesRetainedAfterDifference - 
> org.apache.cassandra.repair.RepairJobTest
> --
>
> Key: CASSANDRA-15872
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15872
> Project: Cassandra
>  Issue Type: Task
>  Components: Consistency/Repair
>Reporter: ZhaoYang
>Priority: Normal
> Fix For: 4.0-rc
>
>
> link: [https://circleci.com/gh/maedhroz/cassandra/21#tests/containers/95]
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-15872) Flaky test: testNoTreesRetainedAfterDifference - org.apache.cassandra.repair.RepairJobTest

2020-06-12 Thread ZhaoYang (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ZhaoYang updated CASSANDRA-15872:
-
Fix Version/s: 4.0-rc

> Flaky test: testNoTreesRetainedAfterDifference - 
> org.apache.cassandra.repair.RepairJobTest
> --
>
> Key: CASSANDRA-15872
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15872
> Project: Cassandra
>  Issue Type: Task
>  Components: Consistency/Repair
>Reporter: ZhaoYang
>Priority: Normal
> Fix For: 4.0-rc
>
>
> link: [https://circleci.com/gh/maedhroz/cassandra/21#tests/containers/95]
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Created] (CASSANDRA-15872) Flaky test: testNoTreesRetainedAfterDifference - org.apache.cassandra.repair.RepairJobTest

2020-06-12 Thread ZhaoYang (Jira)
ZhaoYang created CASSANDRA-15872:


 Summary: Flaky test: testNoTreesRetainedAfterDifference - 
org.apache.cassandra.repair.RepairJobTest
 Key: CASSANDRA-15872
 URL: https://issues.apache.org/jira/browse/CASSANDRA-15872
 Project: Cassandra
  Issue Type: Task
  Components: Consistency/Repair
Reporter: ZhaoYang


link: [https://circleci.com/gh/maedhroz/cassandra/21#tests/containers/95]

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-15871) Cassandra driver first connection get the user's own schema information

2020-06-12 Thread Brandon Williams (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams updated CASSANDRA-15871:
-
Resolution: Invalid
Status: Resolved  (was: Triage Needed)

You need to open a ticket against whichever driver you are using.

> Cassandra driver first  connection get the user's own schema information
> 
>
> Key: CASSANDRA-15871
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15871
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Cluster/Schema, Messaging/Client
>Reporter: maxwellguo
>Priority: Normal
>
> We know that cassandra driver making a conenction with the coordinator node 
> first time , the driver may select all the keyspaces/tables/columns/types 
> from the server and cache the data locally. 
> For different users they may have different tables and types ,so It is not 
> suitable to get all the meta data cached , It is fine to just cache the 
> user's own schema information not all.
>  And doing this is safe and save first time connection resourse.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-15863) Bootstrap resume and TestReplaceAddress fixes

2020-06-12 Thread Brandon Williams (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams updated CASSANDRA-15863:
-
Reviewers: Brandon Williams, Brandon Williams  (was: Brandon Williams)
   Brandon Williams, Brandon Williams  (was: Brandon Williams)
   Status: Review In Progress  (was: Patch Available)

> Bootstrap resume and TestReplaceAddress fixes
> -
>
> Key: CASSANDRA-15863
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15863
> Project: Cassandra
>  Issue Type: Bug
>  Components: Consistency/Bootstrap and Decommission, Test/dtest
>Reporter: Berenguer Blasi
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-alpha
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> This has been 
> [broken|https://ci-cassandra.apache.org/job/Cassandra-trunk/159/testReport/dtest-large.replace_address_test/TestReplaceAddress/test_restart_failed_replace/history/]
>  for ages



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-15834) Bloom filter false positive rate calculation does not take into account true negatives

2020-06-12 Thread Brandon Williams (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams updated CASSANDRA-15834:
-
Reviewers: Brandon Williams

> Bloom filter false positive rate calculation does not take into account true 
> negatives
> --
>
> Key: CASSANDRA-15834
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15834
> Project: Cassandra
>  Issue Type: Bug
>  Components: Observability/JMX
>Reporter: Jaroslaw Grabowski
>Assignee: Jaroslaw Grabowski
>Priority: Normal
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The bloom filter false positive ratio is [currently 
> computed|https://github.com/apache/cassandra/blob/ded62076e7fdfd1cfdcf96447489ea607ca796a0/src/java/org/apache/cassandra/metrics/TableMetrics.java#L738]
>  as:
> {{bf_fp_ratio = false_positive_count / (false_positive_count + 
> true_positive_count)}}
> However, this calculation doesn't take into account true negatives (false 
> negatives never happen on bloom filters).
> In a situation where there are 1000 reads for non existing rows, and there 
> are 10 false positives, the bloom filter false positive ratio will be wrongly 
> calculated as 10/10 = 1.0, while it should be 10/1000 = 0.01.
> We should update the calculation to:
> {{bf_fp_ratio = false_positive_count / #bf_queries}}
> Original jira by [~pauloricardomg]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-15869) MemoryOutputStream overflow on large bloom filter

2020-06-12 Thread Brandon Williams (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams updated CASSANDRA-15869:
-
  Fix Version/s: (was: 4.0-beta)
 4.0-alpha5
  Since Version: 4.0-alpha1
Source Control Link: 
https://github.com/apache/cassandra/commit/7cdad3ce61664cf2ffac75b6e537c910e573754c
  (was: https://github.com/apache/cassandra/pull/620)
 Resolution: Fixed
 Status: Resolved  (was: Ready to Commit)

Thanks, committed.

> MemoryOutputStream overflow on large bloom filter
> -
>
> Key: CASSANDRA-15869
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15869
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/SSTable
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 4.0-alpha5
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Since CASSANDRA-9067, Cassandra will use {{MemoryOutputStream}} to 
> reconstruct BF Memory without re-ordering bytes, see 
> {{OffsetBitSet#deserialize}}. But {{MemoryOutputStream}} use {{INT}} to track 
> position and will overflow when BF size exceeds 2GB.
> {code:java|title=stacktrace}
> error: Illegal bounds [-2147483648..-2147483584); size: 4588588016
> -- StackTrace --
> java.lang.AssertionError: Illegal bounds [-2147483648..-2147483584); size: 
> 4588588016
>   at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:185)
>   at org.apache.cassandra.io.util.Memory.setBytes(Memory.java:138)
>   at 
> org.apache.cassandra.io.util.MemoryOutputStream.write(MemoryOutputStream.java:45)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-15869) MemoryOutputStream overflow on large bloom filter

2020-06-12 Thread Brandon Williams (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams updated CASSANDRA-15869:
-
Reviewers: Brandon Williams, Brandon Williams  (was: Brandon Williams)
   Brandon Williams, Brandon Williams
   Status: Review In Progress  (was: Patch Available)

> MemoryOutputStream overflow on large bloom filter
> -
>
> Key: CASSANDRA-15869
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15869
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/SSTable
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Since CASSANDRA-9067, Cassandra will use {{MemoryOutputStream}} to 
> reconstruct BF Memory without re-ordering bytes, see 
> {{OffsetBitSet#deserialize}}. But {{MemoryOutputStream}} use {{INT}} to track 
> position and will overflow when BF size exceeds 2GB.
> {code:java|title=stacktrace}
> error: Illegal bounds [-2147483648..-2147483584); size: 4588588016
> -- StackTrace --
> java.lang.AssertionError: Illegal bounds [-2147483648..-2147483584); size: 
> 4588588016
>   at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:185)
>   at org.apache.cassandra.io.util.Memory.setBytes(Memory.java:138)
>   at 
> org.apache.cassandra.io.util.MemoryOutputStream.write(MemoryOutputStream.java:45)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-15869) MemoryOutputStream overflow on large bloom filter

2020-06-12 Thread Brandon Williams (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams updated CASSANDRA-15869:
-
Status: Ready to Commit  (was: Review In Progress)

> MemoryOutputStream overflow on large bloom filter
> -
>
> Key: CASSANDRA-15869
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15869
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/SSTable
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Since CASSANDRA-9067, Cassandra will use {{MemoryOutputStream}} to 
> reconstruct BF Memory without re-ordering bytes, see 
> {{OffsetBitSet#deserialize}}. But {{MemoryOutputStream}} use {{INT}} to track 
> position and will overflow when BF size exceeds 2GB.
> {code:java|title=stacktrace}
> error: Illegal bounds [-2147483648..-2147483584); size: 4588588016
> -- StackTrace --
> java.lang.AssertionError: Illegal bounds [-2147483648..-2147483584); size: 
> 4588588016
>   at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:185)
>   at org.apache.cassandra.io.util.Memory.setBytes(Memory.java:138)
>   at 
> org.apache.cassandra.io.util.MemoryOutputStream.write(MemoryOutputStream.java:45)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[cassandra] branch trunk updated: Avoid overflow when bloom filter size exceeds 2GB

2020-06-12 Thread brandonwilliams
This is an automated email from the ASF dual-hosted git repository.

brandonwilliams pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git


The following commit(s) were added to refs/heads/trunk by this push:
 new 7cdad3c  Avoid overflow when bloom filter size exceeds 2GB
7cdad3c is described below

commit 7cdad3ce61664cf2ffac75b6e537c910e573754c
Author: Zhao Yang 
AuthorDate: Thu Jun 11 13:05:47 2020 +0800

Avoid overflow when bloom filter size exceeds 2GB

Patch by Zhao Yang, reviewed by brandonwilliams for CASSANDRA-15869
---
 src/java/org/apache/cassandra/io/util/MemoryOutputStream.java | 7 +--
 1 file changed, 1 insertion(+), 6 deletions(-)

diff --git a/src/java/org/apache/cassandra/io/util/MemoryOutputStream.java 
b/src/java/org/apache/cassandra/io/util/MemoryOutputStream.java
index e6c869d..c984738 100644
--- a/src/java/org/apache/cassandra/io/util/MemoryOutputStream.java
+++ b/src/java/org/apache/cassandra/io/util/MemoryOutputStream.java
@@ -27,7 +27,7 @@ public class MemoryOutputStream extends OutputStream
 {
 
 private final Memory mem;
-private int position = 0;
+private long position = 0;
 
 public MemoryOutputStream(Memory mem)
 {
@@ -45,9 +45,4 @@ public class MemoryOutputStream extends OutputStream
 mem.setBytes(position, b, off, len);
 position += len;
 }
-
-public int position()
-{
-return position;
-}
 }


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Commented] (CASSANDRA-15867) Update Jackson version to 2.9.10.1 because there are security issues in 2.9.5

2020-06-12 Thread Brandon Williams (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-15867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134251#comment-17134251
 ] 

Brandon Williams commented on CASSANDRA-15867:
--

We can merge it into 3.11 as well, your patch was against trunk so I just went 
with that.  The patch doesn't cleanly apply to 3.11.

> Update Jackson version to 2.9.10.1 because there are security issues in 2.9.5
> -
>
> Key: CASSANDRA-15867
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15867
> Project: Cassandra
>  Issue Type: Task
>  Components: Dependencies
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0-alpha5
>
> Attachments: dependency-check-report.html
>
>
> Please see attached HTML report from OWASP dependency check for current 
> 4.0-alpha5 trunk branch.
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Commented] (CASSANDRA-12126) CAS Reads Inconsistencies

2020-06-12 Thread Sylvain Lebresne (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-12126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134239#comment-17134239
 ] 

Sylvain Lebresne commented on CASSANDRA-12126:
--

Ok, I've rebased the patch against 4.0 and started CI on it all:
||branch||CI||
|[3.0|https://github.com/pcmanus/cassandra/tree/C-12126-3.0]|[Run 
#146|https://ci-cassandra.apache.org/job/Cassandra-devbranch/146/]|
|[3.11|https://github.com/pcmanus/cassandra/tree/C-12126-3.11]|[Run 
#147|https://ci-cassandra.apache.org/job/Cassandra-devbranch/147/]|
|[4.0|https://github.com/pcmanus/cassandra/tree/C-12126-4.0]|[Run 
#148|https://ci-cassandra.apache.org/job/Cassandra-devbranch/148/]|

I included a commit to add the flag that disables the new empty commit for 
SERIAL reads as suggested by [~bdeggleston] earlier. Still slightly on the 
fence on the need for such flag, but I call it "unsafe" 
({{-Dcassandra.unsafe.disable-serial-reads-linearizability}} to be specific) 
and log a warning when used, so I'm at peace with that.

I'll note for future reviewers that while the 3.11 branch is almost a straight 
away merge up of 3.0, there is a minor differences on the 4.0 branch, namely:
 * the added in-jvm dtests needed a few changes to reflect 4.0 changes. To make 
that easier, I squashed 2 of the commits from the 3.0/3.11 branches, which is 
why that branch has one less commit.
 * There is a few changes related to the translation of 
{{WriteTimeoutException}} into {{CasWriteTimeoutException}} (I pushed it down 
in some cases). I believe this fixes a minor "bug" where the "contentions" 
number we returned with {{CasWriteTimeoutException}} was potentially inaccurate 
(namely, if we timed out in {{beginRepairAndPaxos}}, contention leading to that 
exception would be ignored)

I'll wait on getting usable CI results to officially mark it 'ready to review', 
but it is in spirit if anyone is burning to look at this.

> CAS Reads Inconsistencies 
> --
>
> Key: CASSANDRA-12126
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12126
> Project: Cassandra
>  Issue Type: Bug
>  Components: Feature/Lightweight Transactions, Legacy/Coordination
>Reporter: Sankalp Kohli
>Assignee: Sylvain Lebresne
>Priority: Normal
>  Labels: LWT, pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> While looking at the CAS code in Cassandra, I found a potential issue with 
> CAS Reads. Here is how it can happen with RF=3
> 1) You issue a CAS Write and it fails in the propose phase. A machine replies 
> true to a propose and saves the commit in accepted filed. The other two 
> machines B and C does not get to the accept phase. 
> Current state is that machine A has this commit in paxos table as accepted 
> but not committed and B and C does not. 
> 2) Issue a CAS Read and it goes to only B and C. You wont be able to read the 
> value written in step 1. This step is as if nothing is inflight. 
> 3) Issue another CAS Read and it goes to A and B. Now we will discover that 
> there is something inflight from A and will propose and commit it with the 
> current ballot. Now we can read the value written in step 1 as part of this 
> CAS read.
> If we skip step 3 and instead run step 4, we will never learn about value 
> written in step 1. 
> 4. Issue a CAS Write and it involves only B and C. This will succeed and 
> commit a different value than step 1. Step 1 value will never be seen again 
> and was never seen before. 
> If you read the Lamport “paxos made simple” paper and read section 2.3. It 
> talks about this issue which is how learners can find out if majority of the 
> acceptors have accepted the proposal. 
> In step 3, it is correct that we propose the value again since we dont know 
> if it was accepted by majority of acceptors. When we ask majority of 
> acceptors, and more than one acceptors but not majority has something in 
> flight, we have no way of knowing if it is accepted by majority of acceptors. 
> So this behavior is correct. 
> However we need to fix step 2, since it caused reads to not be linearizable 
> with respect to writes and other reads. In this case, we know that majority 
> of acceptors have no inflight commit which means we have majority that 
> nothing was accepted by majority. I think we should run a propose step here 
> with empty commit and that will cause write written in step 1 to not be 
> visible ever after. 
> With this fix, we will either see data written in step 1 on next serial read 
> or will never see it which is what we want. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Commented] (CASSANDRA-15752) Range read concurrency factor didn't consider range merger

2020-06-12 Thread ZhaoYang (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-15752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134124#comment-17134124
 ] 

ZhaoYang commented on CASSANDRA-15752:
--

updated the branch and rebased on trunk

> Range read concurrency factor didn't consider range merger
> --
>
> Key: CASSANDRA-15752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15752
> Project: Cassandra
>  Issue Type: Bug
>  Components: Legacy/Coordination
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
>
> During range read, coordinator computes concurrency factor which is the 
> number of vnode ranges to contact in parallel for the next batch.
> But in {{RangeCommandIterator}}, vnode ranges are merged by {{RangeMerger}} 
> if vnode ranges share enough replicas to satisfy consistency level. eg. vnode 
> range [a,b) has replica n1,n2,n3 and vnode range [b,c) has replica n2,n3,n4, 
> so they can be merged as range [a,c) with replica n2, n3 for Quorum.
> Currently it counts number of merged ranges towards concurrency factor. 
> Coordinator may fetch more ranges than needed.
> 
> Another issue is that when executing range read on table with very small 
> amount of data, concurrency factor can be bumped to {{size of total vnode 
> ranges}}, eg. 10k, depending on the num of vnodes and cluster size. As a 
> result, coordinator will send large number of concurrent range requests, 
> potentially slowing down the cluster.. We should cap the max concurrency 
> factor..



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-15862) Use "allow list" or "safe list" instead of the term "whitelist"

2020-06-12 Thread Sam Tunnicliffe (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe updated CASSANDRA-15862:

  Fix Version/s: (was: 4.0-alpha)
 (was: 3.11.x)
 (was: 3.0.x)
 (was: 2.2.x)
 4.0-alpha5
 3.11.7
 3.0.21
 2.2.17
Source Control Link: 
https://github.com/apache/cassandra/commit/c8c3c269bf1df944b73e2d4bf0008a6f412bf121
 Resolution: Fixed
 Status: Resolved  (was: Ready to Commit)

Thanks, committed to 2.2 in {{c8c3c269bf1df944b73e2d4bf0008a6f412bf121}} and 
merged to trunk. I've added a deprecation notice to {{NEWS.txt}} in trunk only.

> Use "allow list" or "safe list" instead of the term "whitelist" 
> 
>
> Key: CASSANDRA-15862
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15862
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Legacy/Core
>Reporter: Ash Berlin-Taylor
>Assignee: Sam Tunnicliffe
>Priority: Normal
> Fix For: 2.2.17, 3.0.21, 3.11.7, 4.0-alpha5
>
>
> Language matters. I'd like to remove all references in Apache Airflow to 
> whitelist or black list, and the Cassandra Python API has some that we can't 
> easily remove.
> The recent global events have made this even more relevant, but this has been 
> on my radar for a while now. Here is a well written article for why I think 
> it matters 
> https://www.ncsc.gov.uk/blog-post/terminology-its-not-black-and-white
> {quote}It's fairly common to say whitelisting and blacklisting to describe 
> desirable and undesirable things in cyber security.
> However, there's an issue with the terminology. It only makes sense if you 
> equate white with 'good, permitted, safe' and black with 'bad, dangerous, 
> forbidden'. There are some obvious problems with this. {quote}
> My exposure to is via the Python API where there is the 
> cassandra.pollicies.WhiteListRoundRobinPolicy class. I propose that this be 
> renamed to AllowListRoundRobinPolicy instead. I do not know if there are 
> other references.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-15862) Use "allow list" or "safe list" instead of the term "whitelist"

2020-06-12 Thread Sam Tunnicliffe (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe updated CASSANDRA-15862:

Status: Ready to Commit  (was: Review In Progress)

> Use "allow list" or "safe list" instead of the term "whitelist" 
> 
>
> Key: CASSANDRA-15862
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15862
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Legacy/Core
>Reporter: Ash Berlin-Taylor
>Assignee: Sam Tunnicliffe
>Priority: Normal
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.0-alpha
>
>
> Language matters. I'd like to remove all references in Apache Airflow to 
> whitelist or black list, and the Cassandra Python API has some that we can't 
> easily remove.
> The recent global events have made this even more relevant, but this has been 
> on my radar for a while now. Here is a well written article for why I think 
> it matters 
> https://www.ncsc.gov.uk/blog-post/terminology-its-not-black-and-white
> {quote}It's fairly common to say whitelisting and blacklisting to describe 
> desirable and undesirable things in cyber security.
> However, there's an issue with the terminology. It only makes sense if you 
> equate white with 'good, permitted, safe' and black with 'bad, dangerous, 
> forbidden'. There are some obvious problems with this. {quote}
> My exposure to is via the Python API where there is the 
> cassandra.pollicies.WhiteListRoundRobinPolicy class. I propose that this be 
> renamed to AllowListRoundRobinPolicy instead. I do not know if there are 
> other references.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[cassandra] 01/01: Merge branch 'cassandra-3.0' into cassandra-3.11

2020-06-12 Thread samt
This is an automated email from the ASF dual-hosted git repository.

samt pushed a commit to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 408f969f3c1f6cccdb26f4ef8daec010b87933b5
Merge: e498539 0a1e8d1
Author: Sam Tunnicliffe 
AuthorDate: Fri Jun 12 11:26:04 2020 +0100

Merge branch 'cassandra-3.0' into cassandra-3.11

 CHANGES.txt|  1 +
 .../cassandra/auth/jmx/AuthorizationProxy.java | 47 +++---
 .../cassandra/cql3/functions/UDFunction.java   | 22 +-
 src/java/org/apache/cassandra/db/Directories.java  | 41 +--
 ...Directories.java => DisallowedDirectories.java} | 29 ++---
 ...sMBean.java => DisallowedDirectoriesMBean.java} |  2 +-
 .../org/apache/cassandra/db/DiskBoundaries.java|  2 +-
 .../apache/cassandra/db/DiskBoundaryManager.java   |  6 +--
 .../db/compaction/AbstractCompactionStrategy.java  |  6 +--
 .../cassandra/db/compaction/LeveledManifest.java   |  2 +-
 .../org/apache/cassandra/dht/RangeStreamer.java| 10 ++---
 .../cassandra/hints/HintsDispatchExecutor.java |  2 +-
 .../org/apache/cassandra/hints/HintsStore.java | 10 ++---
 .../cassandra/io/sstable/format/SSTableReader.java |  2 +-
 .../cassandra/service/DefaultFSErrorHandler.java   |  6 +--
 .../apache/cassandra/service/StorageService.java   |  2 +-
 test/unit/org/apache/cassandra/Util.java   |  4 +-
 .../cassandra/auth/jmx/AuthorizationProxyTest.java |  4 +-
 .../org/apache/cassandra/db/DirectoriesTest.java   |  2 +-
 .../cassandra/db/DiskBoundaryManagerTest.java  |  6 +--
 java => CorruptedSSTablesCompactionsTest.java} | 17 
 21 files changed, 112 insertions(+), 111 deletions(-)

diff --cc src/java/org/apache/cassandra/auth/jmx/AuthorizationProxy.java
index 7bfbf52,000..ebc1763
mode 100644,00..100644
--- a/src/java/org/apache/cassandra/auth/jmx/AuthorizationProxy.java
+++ b/src/java/org/apache/cassandra/auth/jmx/AuthorizationProxy.java
@@@ -1,494 -1,0 +1,493 @@@
 +/*
 + * Licensed to the Apache Software Foundation (ASF) under one
 + * or more contributor license agreements.  See the NOTICE file
 + * distributed with this work for additional information
 + * regarding copyright ownership.  The ASF licenses this file
 + * to you under the Apache License, Version 2.0 (the
 + * "License"); you may not use this file except in compliance
 + * with the License.  You may obtain a copy of the License at
 + *
 + * http://www.apache.org/licenses/LICENSE-2.0
 + *
 + * Unless required by applicable law or agreed to in writing, software
 + * distributed under the License is distributed on an "AS IS" BASIS,
 + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 + * See the License for the specific language governing permissions and
 + * limitations under the License.
 + */
 +
 +package org.apache.cassandra.auth.jmx;
 +
 +import java.lang.reflect.*;
 +import java.security.AccessControlContext;
 +import java.security.AccessController;
 +import java.security.Principal;
 +import java.util.Set;
 +import java.util.function.Function;
 +import java.util.function.Supplier;
 +import java.util.stream.Collectors;
 +import javax.management.MBeanServer;
 +import javax.management.MalformedObjectNameException;
 +import javax.management.ObjectName;
 +import javax.security.auth.Subject;
 +
 +import com.google.common.annotations.VisibleForTesting;
- import com.google.common.base.Throwables;
 +import com.google.common.collect.ImmutableSet;
 +import org.slf4j.Logger;
 +import org.slf4j.LoggerFactory;
 +
 +import org.apache.cassandra.auth.*;
 +import org.apache.cassandra.config.DatabaseDescriptor;
 +import org.apache.cassandra.service.StorageService;
 +
 +/**
 + * Provides a proxy interface to the platform's MBeanServer instance to 
perform
 + * role-based authorization on method invocation.
 + *
 + * When used in conjunction with a suitable JMXAuthenticator, which attaches 
a CassandraPrincipal
 + * to authenticated Subjects, this class uses the configured IAuthorizer to 
verify that the
 + * subject has the required permissions to execute methods on the MBeanServer 
and the MBeans it
 + * manages.
 + *
 + * Because an ObjectName may contain wildcards, meaning it represents a set 
of individual MBeans,
 + * JMX resources don't fit well with the hierarchical approach modelled by 
other IResource
 + * implementations and utilised by ClientState::ensureHasPermission etc. To 
enable grants to use
 + * pattern-type ObjectNames, this class performs its own custom matching and 
filtering of resources
 + * rather than pushing that down to the configured IAuthorizer. To that end, 
during authorization
 + * it pulls back all permissions for the active subject, filtering them to 
retain only grants on
 + * JMXResources. It then uses ObjectName::apply to assert whether the target 
MBeans are wholly
 + * represented by the resources with permissions. This means that 

[cassandra] branch cassandra-3.0 updated (c092c46 -> 0a1e8d1)

2020-06-12 Thread samt
This is an automated email from the ASF dual-hosted git repository.

samt pushed a change to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from c092c46  Merge branch 'cassandra-2.2' into cassandra-3.0
 new c8c3c26  Fix nomenclature of deny and allow lists
 new 0a1e8d1  Merge branch 'cassandra-2.2' into cassandra-3.0

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|  1 +
 .../cassandra/cql3/functions/UDFunction.java   | 22 
 src/java/org/apache/cassandra/db/Directories.java  | 28 ++---
 ...Directories.java => DisallowedDirectories.java} | 29 +++---
 ...sMBean.java => DisallowedDirectoriesMBean.java} |  3 ++-
 .../db/compaction/AbstractCompactionStrategy.java  |  6 ++---
 .../cassandra/db/compaction/LeveledManifest.java   |  2 +-
 .../cassandra/hints/HintsDispatchExecutor.java |  2 +-
 .../org/apache/cassandra/hints/HintsStore.java | 10 
 .../cassandra/io/sstable/format/SSTableReader.java |  2 +-
 .../cassandra/service/DefaultFSErrorHandler.java   |  6 ++---
 test/unit/org/apache/cassandra/Util.java   |  4 +--
 .../org/apache/cassandra/db/DirectoriesTest.java   |  2 +-
 java => CorruptedSSTablesCompactionsTest.java} | 17 +++--
 14 files changed, 69 insertions(+), 65 deletions(-)
 rename src/java/org/apache/cassandra/db/{BlacklistedDirectories.java => 
DisallowedDirectories.java} (77%)
 rename src/java/org/apache/cassandra/db/{BlacklistedDirectoriesMBean.java => 
DisallowedDirectoriesMBean.java} (95%)
 rename 
test/unit/org/apache/cassandra/db/compaction/{BlacklistingCompactionsTest.java 
=> CorruptedSSTablesCompactionsTest.java} (93%)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[cassandra] branch trunk updated (259c63f -> be572f2)

2020-06-12 Thread samt
This is an automated email from the ASF dual-hosted git repository.

samt pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from 259c63f  update version of Jackson to 2.9.10
 new c8c3c26  Fix nomenclature of deny and allow lists
 new 0a1e8d1  Merge branch 'cassandra-2.2' into cassandra-3.0
 new 408f969  Merge branch 'cassandra-3.0' into cassandra-3.11
 new be572f2  Merge branch 'cassandra-3.11' into trunk

The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|  1 +
 NEWS.txt   |  8 
 .../cassandra/auth/jmx/AuthorizationProxy.java | 46 ++---
 .../cassandra/cql3/functions/UDFunction.java   | 22 +-
 src/java/org/apache/cassandra/db/Directories.java  | 47 +++---
 ...Directories.java => DisallowedDirectories.java} | 29 ++---
 ...sMBean.java => DisallowedDirectoriesMBean.java} |  2 +-
 .../org/apache/cassandra/db/DiskBoundaries.java|  2 +-
 .../apache/cassandra/db/DiskBoundaryManager.java   |  6 +--
 .../db/compaction/AbstractCompactionStrategy.java  |  6 +--
 .../cassandra/db/compaction/LeveledManifest.java   |  2 +-
 .../org/apache/cassandra/dht/RangeStreamer.java| 12 +++---
 .../cassandra/hints/HintsDispatchExecutor.java |  2 +-
 .../org/apache/cassandra/hints/HintsStore.java | 10 ++---
 .../cassandra/io/sstable/format/SSTableReader.java |  2 +-
 .../cassandra/service/DefaultFSErrorHandler.java   |  6 +--
 .../apache/cassandra/service/StorageService.java   |  2 +-
 test/unit/org/apache/cassandra/Util.java   |  4 +-
 .../cassandra/auth/jmx/AuthorizationProxyTest.java |  4 +-
 .../org/apache/cassandra/db/DirectoriesTest.java   |  2 +-
 .../cassandra/db/DiskBoundaryManagerTest.java  |  6 +--
 java => CorruptedSSTablesCompactionsTest.java} | 17 
 22 files changed, 124 insertions(+), 114 deletions(-)
 rename src/java/org/apache/cassandra/db/{BlacklistedDirectories.java => 
DisallowedDirectories.java} (80%)
 rename src/java/org/apache/cassandra/db/{BlacklistedDirectoriesMBean.java => 
DisallowedDirectoriesMBean.java} (96%)
 rename 
test/unit/org/apache/cassandra/db/compaction/{BlacklistingCompactionsTest.java 
=> CorruptedSSTablesCompactionsTest.java} (94%)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[cassandra] 01/01: Merge branch 'cassandra-3.11' into trunk

2020-06-12 Thread samt
This is an automated email from the ASF dual-hosted git repository.

samt pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit be572f27416e9477d5ac996087951ecdcd19e25a
Merge: 259c63f 408f969
Author: Sam Tunnicliffe 
AuthorDate: Fri Jun 12 11:28:03 2020 +0100

Merge branch 'cassandra-3.11' into trunk

 CHANGES.txt|  1 +
 NEWS.txt   |  8 
 .../cassandra/auth/jmx/AuthorizationProxy.java | 46 ++---
 .../cassandra/cql3/functions/UDFunction.java   | 22 +-
 src/java/org/apache/cassandra/db/Directories.java  | 47 +++---
 ...Directories.java => DisallowedDirectories.java} | 29 ++---
 ...sMBean.java => DisallowedDirectoriesMBean.java} |  2 +-
 .../org/apache/cassandra/db/DiskBoundaries.java|  2 +-
 .../apache/cassandra/db/DiskBoundaryManager.java   |  6 +--
 .../db/compaction/AbstractCompactionStrategy.java  |  6 +--
 .../cassandra/db/compaction/LeveledManifest.java   |  2 +-
 .../org/apache/cassandra/dht/RangeStreamer.java| 12 +++---
 .../cassandra/hints/HintsDispatchExecutor.java |  2 +-
 .../org/apache/cassandra/hints/HintsStore.java | 10 ++---
 .../cassandra/io/sstable/format/SSTableReader.java |  2 +-
 .../cassandra/service/DefaultFSErrorHandler.java   |  6 +--
 .../apache/cassandra/service/StorageService.java   |  2 +-
 test/unit/org/apache/cassandra/Util.java   |  4 +-
 .../cassandra/auth/jmx/AuthorizationProxyTest.java |  4 +-
 .../org/apache/cassandra/db/DirectoriesTest.java   |  2 +-
 .../cassandra/db/DiskBoundaryManagerTest.java  |  6 +--
 java => CorruptedSSTablesCompactionsTest.java} | 17 
 22 files changed, 124 insertions(+), 114 deletions(-)

diff --cc CHANGES.txt
index 0c4203b,7f54146..5f09cdc
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -55,7 -15,10 +55,8 @@@ Merged from 3.0
   * Allow selecting static column only when querying static index 
(CASSANDRA-14242)
   * cqlsh return non-zero status when STDIN CQL fails (CASSANDRA-15623)
   * Don't skip sstables in slice queries based only on local min/max/deletion 
timestamp (CASSANDRA-15690)
 - * Memtable memory allocations may deadlock (CASSANDRA-15367)
 - * Run evictFromMembership in GossipStage (CASSANDRA-15592)
  Merged from 2.2:
+  * Fix nomenclature of allow and deny lists (CASSANDRA-15862)
   * Remove generated files from source artifact (CASSANDRA-15849)
   * Remove duplicated tools binaries from tarballs (CASSANDRA-15768)
   * Duplicate results with DISTINCT queries in mixed mode (CASSANDRA-15501)
diff --cc NEWS.txt
index 8f18659,077bd8b..351f80d
--- a/NEWS.txt
+++ b/NEWS.txt
@@@ -116,152 -47,8 +116,160 @@@ New feature
  
  Upgrading
  -
 -- Nothing specific to this release, but please see previous upgrading 
sections,
 -  especially if you are upgrading from 3.0.
 +- Sstables for tables using with a frozen UDT written by C* 3.0 appear as 
corrupted.
 +
 +  Background: The serialization-header in the -Statistics.db sstable 
component contains the type information
 +  of the table columns. C* 3.0 write incorrect type information for 
frozen UDTs by omitting the
 +  "frozen" information. Non-frozen UDTs were introduced by CASSANDRA-7423 
in C* 3.6. Since then, the missing
 +  "frozen" information leads to deserialization issues that result in 
CorruptSSTableExceptions, potentially other
 +  exceptions as well.
 +
 +  As a mitigation, the sstable serialization-headers are rewritten to 
contain the missing "frozen" information for
 +  UDTs once, when an upgrade from C* 3.0 is detected. This migration does 
not touch snapshots or backups.
 +
 +  The sstablescrub tool now performs a check of the sstable 
serialization-header against the schema. A mismatch of
 +  the types in the serialization-header and the schema will cause 
sstablescrub to error out and stop by default.
 +  See the new `-e` option. `-e off` disables the new validation code. `-e 
fix` or `-e fix-only`, e.g.
 +  `sstablescrub -e fix keyspace table`, will validate the 
serialization-header, rewrite the non-frozen UDTs
 +  in the serialzation-header to frozen UDTs, if that matches the schema, 
and continue with scrub.
 +  See `sstablescrub -h`.
 +  (CASSANDRA-15035)
 +- CASSANDRA-13241 lowered the default chunk_lengh_in_kb for compresesd 
tables from
 +  64kb to 16kb. For highly compressible data this can have a noticeable 
impact
 +  on space utilization. You may want to consider manually specifying this 
value.
 +- Additional columns have been added to system_distributed.repair_history,
 +  system_traces.sessions and system_traces.events. As a result select 
queries
 +  against these tables - including queries against tracing tables 
performed
 +  automatically by the drivers and cqlsh - will fail and generate an 
error in the log
 +  during upgrade 

[cassandra] branch cassandra-3.11 updated (e498539 -> 408f969)

2020-06-12 Thread samt
This is an automated email from the ASF dual-hosted git repository.

samt pushed a change to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from e498539  Merge branch 'cassandra-3.0' into cassandra-3.11
 new c8c3c26  Fix nomenclature of deny and allow lists
 new 0a1e8d1  Merge branch 'cassandra-2.2' into cassandra-3.0
 new 408f969  Merge branch 'cassandra-3.0' into cassandra-3.11

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|  1 +
 .../cassandra/auth/jmx/AuthorizationProxy.java | 47 +++---
 .../cassandra/cql3/functions/UDFunction.java   | 22 +-
 src/java/org/apache/cassandra/db/Directories.java  | 41 +--
 ...Directories.java => DisallowedDirectories.java} | 29 ++---
 ...sMBean.java => DisallowedDirectoriesMBean.java} |  2 +-
 .../org/apache/cassandra/db/DiskBoundaries.java|  2 +-
 .../apache/cassandra/db/DiskBoundaryManager.java   |  6 +--
 .../db/compaction/AbstractCompactionStrategy.java  |  6 +--
 .../cassandra/db/compaction/LeveledManifest.java   |  2 +-
 .../org/apache/cassandra/dht/RangeStreamer.java| 10 ++---
 .../cassandra/hints/HintsDispatchExecutor.java |  2 +-
 .../org/apache/cassandra/hints/HintsStore.java | 10 ++---
 .../cassandra/io/sstable/format/SSTableReader.java |  2 +-
 .../cassandra/service/DefaultFSErrorHandler.java   |  6 +--
 .../apache/cassandra/service/StorageService.java   |  2 +-
 test/unit/org/apache/cassandra/Util.java   |  4 +-
 .../cassandra/auth/jmx/AuthorizationProxyTest.java |  4 +-
 .../org/apache/cassandra/db/DirectoriesTest.java   |  2 +-
 .../cassandra/db/DiskBoundaryManagerTest.java  |  6 +--
 java => CorruptedSSTablesCompactionsTest.java} | 17 
 21 files changed, 112 insertions(+), 111 deletions(-)
 rename src/java/org/apache/cassandra/db/{BlacklistedDirectories.java => 
DisallowedDirectories.java} (80%)
 rename src/java/org/apache/cassandra/db/{BlacklistedDirectoriesMBean.java => 
DisallowedDirectoriesMBean.java} (96%)
 rename 
test/unit/org/apache/cassandra/db/compaction/{BlacklistingCompactionsTest.java 
=> CorruptedSSTablesCompactionsTest.java} (94%)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[cassandra] 01/01: Merge branch 'cassandra-2.2' into cassandra-3.0

2020-06-12 Thread samt
This is an automated email from the ASF dual-hosted git repository.

samt pushed a commit to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 0a1e8d168bac0f562774814a808e313e1d2d6571
Merge: c092c46 c8c3c26
Author: Sam Tunnicliffe 
AuthorDate: Fri Jun 12 11:21:14 2020 +0100

Merge branch 'cassandra-2.2' into cassandra-3.0

 CHANGES.txt|  1 +
 .../cassandra/cql3/functions/UDFunction.java   | 22 
 src/java/org/apache/cassandra/db/Directories.java  | 28 ++---
 ...Directories.java => DisallowedDirectories.java} | 29 +++---
 ...sMBean.java => DisallowedDirectoriesMBean.java} |  3 ++-
 .../db/compaction/AbstractCompactionStrategy.java  |  6 ++---
 .../cassandra/db/compaction/LeveledManifest.java   |  2 +-
 .../cassandra/hints/HintsDispatchExecutor.java |  2 +-
 .../org/apache/cassandra/hints/HintsStore.java | 10 
 .../cassandra/io/sstable/format/SSTableReader.java |  2 +-
 .../cassandra/service/DefaultFSErrorHandler.java   |  6 ++---
 test/unit/org/apache/cassandra/Util.java   |  4 +--
 .../org/apache/cassandra/db/DirectoriesTest.java   |  2 +-
 java => CorruptedSSTablesCompactionsTest.java} | 17 +++--
 14 files changed, 69 insertions(+), 65 deletions(-)

diff --cc CHANGES.txt
index 3fdbb96,b10a057..d506dc8
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,19 -1,5 +1,20 @@@
 -2.2.18
 +3.0.21
 + * Fix replica-side filtering returning stale data with CL > ONE 
(CASSANDRA-8272, CASSANDRA-8273)
 + * Fix duplicated row on 2.x upgrades when multi-rows range tombstones 
interact with collection ones (CASSANDRA-15805)
 + * Rely on snapshotted session infos on StreamResultFuture.maybeComplete to 
avoid race conditions (CASSANDRA-15667)
 + * EmptyType doesn't override writeValue so could attempt to write bytes when 
expected not to (CASSANDRA-15790)
 + * Fix index queries on partition key columns when some partitions contains 
only static data (CASSANDRA-13666)
 + * Avoid creating duplicate rows during major upgrades (CASSANDRA-15789)
 + * liveDiskSpaceUsed and totalDiskSpaceUsed get corrupted if 
IndexSummaryRedistribution gets interrupted (CASSANDRA-15674)
 + * Fix Debian init start/stop (CASSANDRA-15770)
 + * Fix infinite loop on index query paging in tables with clustering 
(CASSANDRA-14242)
 + * Fix chunk index overflow due to large sstable with small chunk length 
(CASSANDRA-15595)
 + * cqlsh return non-zero status when STDIN CQL fails (CASSANDRA-15623)
 + * Don't skip sstables in slice queries based only on local min/max/deletion 
timestamp (CASSANDRA-15690)
 + * Memtable memory allocations may deadlock (CASSANDRA-15367)
 + * Run evictFromMembership in GossipStage (CASSANDRA-15592)
 +Merged from 2.2:
+  * Fix nomenclature of allow and deny lists (CASSANDRA-15862)
   * Remove generated files from source artifact (CASSANDRA-15849)
   * Remove duplicated tools binaries from tarballs (CASSANDRA-15768)
   * Duplicate results with DISTINCT queries in mixed mode (CASSANDRA-15501)
diff --cc src/java/org/apache/cassandra/cql3/functions/UDFunction.java
index 7b69342,1e5cea6..27f9eb8
--- a/src/java/org/apache/cassandra/cql3/functions/UDFunction.java
+++ b/src/java/org/apache/cassandra/cql3/functions/UDFunction.java
@@@ -71,111 -49,10 +71,111 @@@ public abstract class UDFunction extend
  protected final String language;
  protected final String body;
  
 -protected final DataType[] argDataTypes;
 -protected final DataType returnDataType;
 +protected final TypeCodec[] argCodecs;
 +protected final TypeCodec returnCodec;
  protected final boolean calledOnNullInput;
  
 +//
- // Access to classes is controlled via a whitelist and a blacklist.
++// Access to classes is controlled via allow and disallow lists.
 +//
 +// When a class is requested (both during compilation and runtime),
- // the whitelistedPatterns array is searched first, whether the
++// the allowedPatterns array is searched first, whether the
 +// requested name matches one of the patterns. If not, nothing is
 +// returned from the class-loader - meaning ClassNotFoundException
 +// during runtime and "type could not resolved" during compilation.
 +//
- // If a whitelisted pattern has been found, the blacklistedPatterns
++// If an allowed pattern has been found, the disallowedPatterns
 +// array is searched for a match. If a match is found, class-loader
 +// rejects access. Otherwise the class/resource can be loaded.
 +//
- private static final String[] whitelistedPatterns =
++private static final String[] allowedPatterns =
 +{
 +"com/datastax/driver/core/",
 +"com/google/common/reflect/TypeToken",
 +"java/io/IOException.class",
 +"java/io/Serializable.class",
 +"java/lang/",
 +"java/math/",
 +"java/net/InetAddress.class",
 +"java/net/Inet4Address.class",
 +

[cassandra] branch cassandra-2.2 updated: Fix nomenclature of deny and allow lists

2020-06-12 Thread samt
This is an automated email from the ASF dual-hosted git repository.

samt pushed a commit to branch cassandra-2.2
in repository https://gitbox.apache.org/repos/asf/cassandra.git


The following commit(s) were added to refs/heads/cassandra-2.2 by this push:
 new c8c3c26  Fix nomenclature of deny and allow lists
c8c3c26 is described below

commit c8c3c269bf1df944b73e2d4bf0008a6f412bf121
Author: Sam Tunnicliffe 
AuthorDate: Thu Jun 11 18:40:18 2020 +0100

Fix nomenclature of deny and allow lists

Patch by Sam Tunnicliffe; reviewed by Jordan West for CASSANDRA-15862
---
 CHANGES.txt|  1 +
 src/java/org/apache/cassandra/db/Directories.java  | 24 +-
 ...Directories.java => DisallowedDirectories.java} | 29 +++---
 ...sMBean.java => DisallowedDirectoriesMBean.java} |  3 ++-
 .../db/compaction/AbstractCompactionStrategy.java  |  6 ++---
 .../cassandra/db/compaction/LeveledManifest.java   |  2 +-
 .../cassandra/io/sstable/format/SSTableReader.java |  2 +-
 .../cassandra/service/DefaultFSErrorHandler.java   |  6 ++---
 .../org/apache/cassandra/cql3/OutOfSpaceBase.java  |  4 +--
 .../org/apache/cassandra/db/DirectoriesTest.java   |  2 +-
 java => CorruptedSSTablesCompactionsTest.java} | 15 ++-
 11 files changed, 48 insertions(+), 46 deletions(-)

diff --git a/CHANGES.txt b/CHANGES.txt
index 788b2bf..b10a057 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.18
+ * Fix nomenclature of allow and deny lists (CASSANDRA-15862)
  * Remove generated files from source artifact (CASSANDRA-15849)
  * Remove duplicated tools binaries from tarballs (CASSANDRA-15768)
  * Duplicate results with DISTINCT queries in mixed mode (CASSANDRA-15501)
diff --git a/src/java/org/apache/cassandra/db/Directories.java 
b/src/java/org/apache/cassandra/db/Directories.java
index fa76b61..d1aa650 100644
--- a/src/java/org/apache/cassandra/db/Directories.java
+++ b/src/java/org/apache/cassandra/db/Directories.java
@@ -292,10 +292,10 @@ public class Directories
 
 /**
  * Basically the same as calling {@link #getWriteableLocationAsFile(long)} 
with an unknown size ({@code -1L}),
- * which may return any non-blacklisted directory - even a data directory 
that has no usable space.
+ * which may return any allowed directory - even a data directory that has 
no usable space.
  * Do not use this method in production code.
  *
- * @throws IOError if all directories are blacklisted.
+ * @throws IOError if all directories are blocked.
  */
 public File getDirectoryForNewSSTables()
 {
@@ -303,9 +303,9 @@ public class Directories
 }
 
 /**
- * Returns a non-blacklisted data directory that _currently_ has {@code 
writeSize} bytes as usable space.
+ * Returns an allowed directory that _currently_ has {@code writeSize} 
bytes as usable space.
  *
- * @throws IOError if all directories are blacklisted.
+ * @throws IOError if all directories are disallowed.
  */
 public File getWriteableLocationAsFile(long writeSize)
 {
@@ -313,9 +313,9 @@ public class Directories
 }
 
 /**
- * Returns a non-blacklisted data directory that _currently_ has {@code 
writeSize} bytes as usable space.
+ * Returns an allowed data directory that _currently_ has {@code 
writeSize} bytes as usable space.
  *
- * @throws IOError if all directories are blacklisted.
+ * @throws IOError if all directories are disallowed.
  */
 public DataDirectory getWriteableLocation(long writeSize)
 {
@@ -323,13 +323,13 @@ public class Directories
 
 long totalAvailable = 0L;
 
-// pick directories with enough space and so that resulting sstable 
dirs aren't blacklisted for writes.
+// pick directories with enough space and so that resulting sstable 
dirs aren't disallowed for writes.
 boolean tooBig = false;
 for (DataDirectory dataDir : dataDirectories)
 {
-if 
(BlacklistedDirectories.isUnwritable(getLocationForDisk(dataDir)))
+if 
(DisallowedDirectories.isUnwritable(getLocationForDisk(dataDir)))
 {
-logger.trace("removing blacklisted candidate {}", 
dataDir.location);
+logger.trace("removing disallowed candidate {}", 
dataDir.location);
 continue;
 }
 DataDirectoryCandidate candidate = new 
DataDirectoryCandidate(dataDir);
@@ -348,7 +348,7 @@ public class Directories
 if (tooBig)
 throw new RuntimeException("Insufficient disk space to write " 
+ writeSize + " bytes");
 else
-throw new FSWriteError(new IOException("All configured data 
directories have been blacklisted as unwritable for erroring out"), "");
+throw new FSWriteError(new IOException("All configured data 
directories have been disallowed as unwritable for erroring out"), "");
 
 

[jira] [Created] (CASSANDRA-15871) Cassandra driver first connection get the user's own schema information

2020-06-12 Thread maxwellguo (Jira)
maxwellguo created CASSANDRA-15871:
--

 Summary: Cassandra driver first  connection get the user's own 
schema information
 Key: CASSANDRA-15871
 URL: https://issues.apache.org/jira/browse/CASSANDRA-15871
 Project: Cassandra
  Issue Type: Improvement
  Components: Cluster/Schema, Messaging/Client
Reporter: maxwellguo


We know that cassandra driver making a conenction with the coordinator node 
first time , the driver may select all the keyspaces/tables/columns/types from 
the server and cache the data locally. 
For different users they may have different tables and types ,so It is not 
suitable to get all the meta data cached , It is fine to just cache the user's 
own schema information not all.

 And doing this is safe and save first time connection resourse.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Commented] (CASSANDRA-15869) MemoryOutputStream overflow on large bloom filter

2020-06-12 Thread ZhaoYang (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-15869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134016#comment-17134016
 ] 

ZhaoYang commented on CASSANDRA-15869:
--

[~brandon.williams] do you mind reviewing?

> MemoryOutputStream overflow on large bloom filter
> -
>
> Key: CASSANDRA-15869
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15869
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/SSTable
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Since CASSANDRA-9067, Cassandra will use {{MemoryOutputStream}} to 
> reconstruct BF Memory without re-ordering bytes, see 
> {{OffsetBitSet#deserialize}}. But {{MemoryOutputStream}} use {{INT}} to track 
> position and will overflow when BF size exceeds 2GB.
> {code:java|title=stacktrace}
> error: Illegal bounds [-2147483648..-2147483584); size: 4588588016
> -- StackTrace --
> java.lang.AssertionError: Illegal bounds [-2147483648..-2147483584); size: 
> 4588588016
>   at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:185)
>   at org.apache.cassandra.io.util.Memory.setBytes(Memory.java:138)
>   at 
> org.apache.cassandra.io.util.MemoryOutputStream.write(MemoryOutputStream.java:45)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Updated] (CASSANDRA-15752) Range read concurrency factor didn't consider range merger

2020-06-12 Thread ZhaoYang (Jira)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ZhaoYang updated CASSANDRA-15752:
-
Test and Documentation Plan: 
Added unit tests.

Circle Ci: 
[https://circleci.com/workflow-run/440126c4-058d-4511-8301-16df2bddf5db]
  
 

  was:
Added unit tests.

Circle Ci: 
[https://circleci.com/workflow-run/cfcdbee3-c982-4ff7-ab87-e1b4c36cfdcd]
 


> Range read concurrency factor didn't consider range merger
> --
>
> Key: CASSANDRA-15752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15752
> Project: Cassandra
>  Issue Type: Bug
>  Components: Legacy/Coordination
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
>
> During range read, coordinator computes concurrency factor which is the 
> number of vnode ranges to contact in parallel for the next batch.
> But in {{RangeCommandIterator}}, vnode ranges are merged by {{RangeMerger}} 
> if vnode ranges share enough replicas to satisfy consistency level. eg. vnode 
> range [a,b) has replica n1,n2,n3 and vnode range [b,c) has replica n2,n3,n4, 
> so they can be merged as range [a,c) with replica n2, n3 for Quorum.
> Currently it counts number of merged ranges towards concurrency factor. 
> Coordinator may fetch more ranges than needed.
> 
> Another issue is that when executing range read on table with very small 
> amount of data, concurrency factor can be bumped to {{size of total vnode 
> ranges}}, eg. 10k, depending on the num of vnodes and cluster size. As a 
> result, coordinator will send large number of concurrent range requests, 
> potentially slowing down the cluster.. We should cap the max concurrency 
> factor..



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Commented] (CASSANDRA-15867) Update Jackson version to 2.9.10.1 because there are security issues in 2.9.5

2020-06-12 Thread Stefan Miklosovic (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-15867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17133974#comment-17133974
 ] 

Stefan Miklosovic commented on CASSANDRA-15867:
---

[~brandon.williams] dont we want to see this merged as well in 3.11? I see that 
coordinates for jackson are totally different so maybe I ll do that scan 
against 3.11 and see how we stand. 

> Update Jackson version to 2.9.10.1 because there are security issues in 2.9.5
> -
>
> Key: CASSANDRA-15867
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15867
> Project: Cassandra
>  Issue Type: Task
>  Components: Dependencies
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0-alpha5
>
> Attachments: dependency-check-report.html
>
>
> Please see attached HTML report from OWASP dependency check for current 
> 4.0-alpha5 trunk branch.
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org