[jira] [Created] (CASSANDRA-15874) Bootstrap completes Successfully without streaming all the data
Jai Bheemsen Rao Dhanwada created CASSANDRA-15874:
-
Summary: Bootstrap completes Successfully without streaming all
the data
Key: CASSANDRA-15874
URL: https://issues.apache.org/jira/browse/CASSANDRA-15874
Project: Cassandra
Issue Type: Bug
Components: Consistency/Bootstrap and Decommission
Reporter: Jai Bheemsen Rao Dhanwada
I am seeing a strange issue where, adding a new node with auto_bootstrap: true
is not streaming all the data before it joins the cluster. Don't see any
information in the logs about bootstrap failures.
Here is the sequence of logs
{code:java}
INFO [main] 2020-06-12 01:41:49,642 StorageService.java:1446 - JOINING: schema
complete, ready to bootstrap
INFO [main] 2020-06-12 01:41:49,642 StorageService.java:1446 - JOINING: waiting
for pending range calculation
INFO [main] 2020-06-12 01:41:49,643 StorageService.java:1446 - JOINING:
calculation complete, ready to bootstrap
INFO [main] 2020-06-12 01:41:49,643 StorageService.java:1446 - JOINING: getting
bootstrap token
INFO [main] 2020-06-12 01:42:19,656 StorageService.java:1446 - JOINING:
Starting to bootstrap...
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for
cfId . If a table was just created, this is likely due to the schema not
being fully propagated. Please wait for schema agreement on table creation.
INFO [StreamReceiveTask:1] 2020-06-12 02:29:51,892 StreamResultFuture.java:219
- [Stream #f4224f444-a55d-154a-23e3-867899486f5f] All sessions completed INFO
[StreamReceiveTask:1] 2020-06-12 02:29:51,892 StorageService.java:1505 -
Bootstrap completed! for the tokens
{code}
Cassandra Version: 3.11.3
I am not able to reproduce this issue all the time, but it happened couple of
times. Is there any race condition/corner case, which could cause this issue?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15794) Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact tables in downgraded C* (3.11.4) introduces non-existent columns
[
https://issues.apache.org/jira/browse/CASSANDRA-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134575#comment-17134575
]
Zhuqi Jin commented on CASSANDRA-15794:
---
Hi, [~ifesdjeen].
I created patches for 3.0 and 3.11, according to the third method we discussed
earlier.
[^CASSANDRA-15794-branch-3.0.patch]
I've attached the patches. Would you mind reviewing them?
> Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact
> tables in downgraded C* (3.11.4) introduces non-existent columns
> -
>
> Key: CASSANDRA-15794
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15794
> Project: Cassandra
> Issue Type: Bug
>Reporter: Zhuqi Jin
>Priority: Normal
> Attachments: CASSANDRA-15794-branch-3.0.patch,
> CASSANDRA-15794-branch-3.11.patch
>
>
> We tried to test upgrading a 3.11.4 C* cluster to 4.x and run into the
> following problems.
> * We started a single 3.11.4 C* node.
> * We ran cassandra-stress like this
> {code:java}
> ./cassandra-stress write n = 30 -rate threads = 10 -node 172.17.0.2 {code}
> * We stopped this node, and started a C* node running C* compiled from trunk
> (git commit: e394dc0bb32f612a476269010930c617dd1ed3cb)
> * New C* failed to start with the following error message
> {code:java}
> ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 - Error while
> loading schema: ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245
> - Error while loading schema: java.lang.IllegalArgumentException: Compact
> Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ...
> DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to
> migrate off Compact Storage. at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862)
> at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at
> org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770)Exception
> (java.lang.IllegalArgumentException) encountered during startup: Compact
> Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ...
> DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to
> migrate off Compact Storage.ERROR [main] 2020-05-07 00:58:18,520
> CassandraDaemon.java:792 - Exception encountered during
> startupjava.lang.IllegalArgumentException: Compact Tables are not allowed in
> Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE`
> command supplied in 3.x/3.11 Cassandra in order to migrate off Compact
> Storage. at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862)
> at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at
> org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770){code}
> * We stopped the trunk version C* and started the 3.11.4 version C*.
> * 3.11.4 C* failed to start with the following error messages:
> {code:java}
> Exception (java.lang.IllegalStateException) encountered during startup:
> Unknown commitlog version 7Exception (java.lang.IllegalStateException)
> encountered during startup: Unknown commitlog version 7ERROR [main]
> 2020-05-07 01:13:26,798 CassandraDaemon.java:749 - Exception encountered
> during startupjava.lang.IllegalStateException: Unknown commitlog version 7 at
> org.apache.cassandra.db.commitlog.CommitLogDescriptor.getMessagingVersion(CommitLogDescriptor.java:227)
> ~[main/:na] at
>
[jira] [Updated] (CASSANDRA-15794) Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact tables in downgraded C* (3.11.4) introduces non-existent columns
[
https://issues.apache.org/jira/browse/CASSANDRA-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhuqi Jin updated CASSANDRA-15794:
--
Attachment: CASSANDRA-15794-branch-3.11.patch
CASSANDRA-15794-branch-3.0.patch
> Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact
> tables in downgraded C* (3.11.4) introduces non-existent columns
> -
>
> Key: CASSANDRA-15794
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15794
> Project: Cassandra
> Issue Type: Bug
>Reporter: Zhuqi Jin
>Priority: Normal
> Attachments: CASSANDRA-15794-branch-3.0.patch,
> CASSANDRA-15794-branch-3.11.patch
>
>
> We tried to test upgrading a 3.11.4 C* cluster to 4.x and run into the
> following problems.
> * We started a single 3.11.4 C* node.
> * We ran cassandra-stress like this
> {code:java}
> ./cassandra-stress write n = 30 -rate threads = 10 -node 172.17.0.2 {code}
> * We stopped this node, and started a C* node running C* compiled from trunk
> (git commit: e394dc0bb32f612a476269010930c617dd1ed3cb)
> * New C* failed to start with the following error message
> {code:java}
> ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 - Error while
> loading schema: ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245
> - Error while loading schema: java.lang.IllegalArgumentException: Compact
> Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ...
> DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to
> migrate off Compact Storage. at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862)
> at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at
> org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770)Exception
> (java.lang.IllegalArgumentException) encountered during startup: Compact
> Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ...
> DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to
> migrate off Compact Storage.ERROR [main] 2020-05-07 00:58:18,520
> CassandraDaemon.java:792 - Exception encountered during
> startupjava.lang.IllegalArgumentException: Compact Tables are not allowed in
> Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE`
> command supplied in 3.x/3.11 Cassandra in order to migrate off Compact
> Storage. at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862)
> at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at
> org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770){code}
> * We stopped the trunk version C* and started the 3.11.4 version C*.
> * 3.11.4 C* failed to start with the following error messages:
> {code:java}
> Exception (java.lang.IllegalStateException) encountered during startup:
> Unknown commitlog version 7Exception (java.lang.IllegalStateException)
> encountered during startup: Unknown commitlog version 7ERROR [main]
> 2020-05-07 01:13:26,798 CassandraDaemon.java:749 - Exception encountered
> during startupjava.lang.IllegalStateException: Unknown commitlog version 7 at
> org.apache.cassandra.db.commitlog.CommitLogDescriptor.getMessagingVersion(CommitLogDescriptor.java:227)
> ~[main/:na] at
> org.apache.cassandra.db.commitlog.CommitLogReader.shouldSkipSegmentId(CommitLogReader.java:276)
> ~[main/:na] at
>
[jira] [Commented] (CASSANDRA-14888) Several mbeans are not unregistered when dropping a keyspace and table
[ https://issues.apache.org/jira/browse/CASSANDRA-14888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134568#comment-17134568 ] Caleb Rackliffe commented on CASSANDRA-14888: - [~stillalex] The unit and in-JVM dtests look pretty good so far. I'm sorting out some hardware issues w/ the dtests, but I'll let you know when those resolve. > Several mbeans are not unregistered when dropping a keyspace and table > -- > > Key: CASSANDRA-14888 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14888 > Project: Cassandra > Issue Type: Bug > Components: Observability/Metrics >Reporter: Ariel Weisberg >Assignee: Alex Deparvu >Priority: Urgent > Labels: patch-available > Fix For: 3.0.x, 3.11.x, 4.0-beta > > Attachments: CASSANDRA-14888.patch > > Time Spent: 2h 20m > Remaining Estimate: 0h > > CasCommit, CasPrepare, CasPropose, ReadRepairRequests, > ShortReadProtectionRequests, AntiCompactionTime, BytesValidated, > PartitionsValidated, RepairPrepareTime, RepairSyncTime, > RepairedDataInconsistencies, ViewLockAcquireTime, ViewReadTime, > WriteFailedIdealCL > Basically for 3 years people haven't known what they are doing because the > entire thing is kind of obscure. Fix it and also add a dtest that detects if > any mbeans are left behind after dropping a table and keyspace. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15873) Update Netty 4.0.44 -> 4.1.50 (fix security/performance issues)
[ https://issues.apache.org/jira/browse/CASSANDRA-15873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134472#comment-17134472 ] Matt Davis commented on CASSANDRA-15873: Works for me, thanks! > Update Netty 4.0.44 -> 4.1.50 (fix security/performance issues) > --- > > Key: CASSANDRA-15873 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15873 > Project: Cassandra > Issue Type: Task > Components: Dependencies >Reporter: Matt Davis >Priority: Normal > Attachments: dependency-check-report.html > > > See https://issues.apache.org/jira/browse/CASSANDRA-15868 for the same issue > on 4.0 / trunk. Attached is an OWASP dependency report for Netty 4.0.44, > which identifies 3 of the same vulnerabilities as above. > > Additionally, 4.1.50 contains aarch64 native libraries which can improve > performance on ARM processors. > > (If the preference is to handle PRs for both versions/branches in a single > issue, feel free to close this as a duplicate.) > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15873) Update Netty 4.0.44 -> 4.1.50 (fix security/performance issues)
[ https://issues.apache.org/jira/browse/CASSANDRA-15873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-15873: - Resolution: Duplicate Status: Resolved (was: Triage Needed) Matt, I updated the fixver for CASSANDRA-15868 to include 3.11. Let's handle everything there. > Update Netty 4.0.44 -> 4.1.50 (fix security/performance issues) > --- > > Key: CASSANDRA-15873 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15873 > Project: Cassandra > Issue Type: Task > Components: Dependencies >Reporter: Matt Davis >Priority: Normal > Attachments: dependency-check-report.html > > > See https://issues.apache.org/jira/browse/CASSANDRA-15868 for the same issue > on 4.0 / trunk. Attached is an OWASP dependency report for Netty 4.0.44, > which identifies 3 of the same vulnerabilities as above. > > Additionally, 4.1.50 contains aarch64 native libraries which can improve > performance on ARM processors. > > (If the preference is to handle PRs for both versions/branches in a single > issue, feel free to close this as a duplicate.) > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15868) Update Netty version to 4.1.50 because there are security issues in 4.1.37
[ https://issues.apache.org/jira/browse/CASSANDRA-15868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-15868: - Fix Version/s: 4.0-beta 3.11.7 > Update Netty version to 4.1.50 because there are security issues in 4.1.37 > -- > > Key: CASSANDRA-15868 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15868 > Project: Cassandra > Issue Type: Task > Components: Dependencies >Reporter: Stefan Miklosovic >Priority: Normal > Fix For: 3.11.7, 4.0-beta > > Attachments: dependency-check-report.html > > > Please see attached HTML report from OWASP dependency check. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15867) Update Jackson version to 2.9.10.1 because there are security issues in 2.9.5
[ https://issues.apache.org/jira/browse/CASSANDRA-15867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134464#comment-17134464 ] Brandon Williams edited comment on CASSANDRA-15867 at 6/12/20, 6:50 PM: I would say if we fix it in one branch, but another is also vulnerable for the same reason, we should fix it there too. bq. This holds for more dependencies, what is the general approach here? I would take it on a case-by-case basis. I looked into the Jackson vulnerability and it does seem to be exploitable for us (though I don't know why users would DoS their database on purpose, certainly accidents can happen.) was (Author: brandon.williams): I would say if we fix it in one branch, but another is also vulnerable for the same reason, we should fix it there too. > This holds for more dependencies, what is the general approach here? I would take it on a case-by-case basis. I looked into the Jackson vulnerability and it does seem to be exploitable for us (though I don't know why users would DoS their database on purpose, certainly accidents can happen.) > Update Jackson version to 2.9.10.1 because there are security issues in 2.9.5 > - > > Key: CASSANDRA-15867 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15867 > Project: Cassandra > Issue Type: Task > Components: Dependencies >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0-alpha5 > > Attachments: dependency-check-report.html > > > Please see attached HTML report from OWASP dependency check for current > 4.0-alpha5 trunk branch. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15867) Update Jackson version to 2.9.10.1 because there are security issues in 2.9.5
[ https://issues.apache.org/jira/browse/CASSANDRA-15867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134464#comment-17134464 ] Brandon Williams commented on CASSANDRA-15867: -- I would say if we fix it in one branch, but another is also vulnerable for the same reason, we should fix it there too. > This holds for more dependencies, what is the general approach here? I would take it on a case-by-case basis. I looked into the Jackson vulnerability and it does seem to be exploitable for us (though I don't know why users would DoS their database on purpose, certainly accidents can happen.) > Update Jackson version to 2.9.10.1 because there are security issues in 2.9.5 > - > > Key: CASSANDRA-15867 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15867 > Project: Cassandra > Issue Type: Task > Components: Dependencies >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0-alpha5 > > Attachments: dependency-check-report.html > > > Please see attached HTML report from OWASP dependency check for current > 4.0-alpha5 trunk branch. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15867) Update Jackson version to 2.9.10.1 because there are security issues in 2.9.5
[ https://issues.apache.org/jira/browse/CASSANDRA-15867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134461#comment-17134461 ] Stefan Miklosovic commented on CASSANDRA-15867: --- [~brandon.williams] I tried to apply it on 3.11 but there are incompatibilies on a source-code levels (classes not there etc). I do not want to make any changes to the codebase of 3.11 unnecessarilly, what is your opinion here? Should I invest time to bump it while it is not so trivial? This holds for more dependencies, what is the general approach here? > Update Jackson version to 2.9.10.1 because there are security issues in 2.9.5 > - > > Key: CASSANDRA-15867 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15867 > Project: Cassandra > Issue Type: Task > Components: Dependencies >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0-alpha5 > > Attachments: dependency-check-report.html > > > Please see attached HTML report from OWASP dependency check for current > 4.0-alpha5 trunk branch. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15867) Update Jackson version to 2.9.10.1 because there are security issues in 2.9.5
[ https://issues.apache.org/jira/browse/CASSANDRA-15867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134461#comment-17134461 ] Stefan Miklosovic edited comment on CASSANDRA-15867 at 6/12/20, 6:43 PM: - [~brandon.williams] I tried to apply it on 3.11 but there are incompatibilies on source-code level (classes not there etc). I do not want to make any changes to the codebase of 3.11 unnecessarilly, what is your opinion here? Should I invest time to bump it while it is not so trivial? This holds for more dependencies, what is the general approach here? was (Author: stefan.miklosovic): [~brandon.williams] I tried to apply it on 3.11 but there are incompatibilies on a source-code levels (classes not there etc). I do not want to make any changes to the codebase of 3.11 unnecessarilly, what is your opinion here? Should I invest time to bump it while it is not so trivial? This holds for more dependencies, what is the general approach here? > Update Jackson version to 2.9.10.1 because there are security issues in 2.9.5 > - > > Key: CASSANDRA-15867 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15867 > Project: Cassandra > Issue Type: Task > Components: Dependencies >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0-alpha5 > > Attachments: dependency-check-report.html > > > Please see attached HTML report from OWASP dependency check for current > 4.0-alpha5 trunk branch. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15868) Update Netty version to 4.1.50 because there are security issues in 4.1.37
[ https://issues.apache.org/jira/browse/CASSANDRA-15868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17130112#comment-17130112 ] Stefan Miklosovic edited comment on CASSANDRA-15868 at 6/12/20, 6:40 PM: - [https://github.com/apache/cassandra/pull/631/files] was (Author: stefan.miklosovic): [https://github.com/apache/cassandra/compare/trunk...smiklosovic:CASSANDRA-15868?expand=1] > Update Netty version to 4.1.50 because there are security issues in 4.1.37 > -- > > Key: CASSANDRA-15868 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15868 > Project: Cassandra > Issue Type: Task > Components: Dependencies >Reporter: Stefan Miklosovic >Priority: Normal > Attachments: dependency-check-report.html > > > Please see attached HTML report from OWASP dependency check. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15868) Update Netty version to 4.1.50 because there are security issues in 4.1.37
[ https://issues.apache.org/jira/browse/CASSANDRA-15868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134454#comment-17134454 ] Matt Davis commented on CASSANDRA-15868: Note also that 4.1.50 contains aarch64 native libraries for improved performance on ARM processors. See https://issues.apache.org/jira/browse/CASSANDRA-15873 for the same issue on Cassandra 3.11 (Netty 4.0.44 -> 4.1.50). > Update Netty version to 4.1.50 because there are security issues in 4.1.37 > -- > > Key: CASSANDRA-15868 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15868 > Project: Cassandra > Issue Type: Task > Components: Dependencies >Reporter: Stefan Miklosovic >Priority: Normal > Attachments: dependency-check-report.html > > > Please see attached HTML report from OWASP dependency check. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15873) Update Netty 4.0.44 -> 4.1.50 (fix security/performance issues)
[ https://issues.apache.org/jira/browse/CASSANDRA-15873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Davis updated CASSANDRA-15873: --- Description: See https://issues.apache.org/jira/browse/CASSANDRA-15868 for the same issue on 4.0 / trunk. Attached is an OWASP dependency report for Netty 4.0.44, which identifies 3 of the same vulnerabilities as above. Additionally, 4.1.50 contains aarch64 native libraries which can improve performance on ARM processors. (If the preference is to handle PRs for both versions/branches in a single issue, feel free to close this as a duplicate.) was: See https://issues.apache.org/jira/browse/CASSANDRA-15868 for the same issue on 4.0 / trunk. Attached is an OWASP dependency report for Netty 4.0.44, which identifies 3 of the same vulnerabilities as the above issue. Additionally, 4.1.50 contains aarch64 native libraries which can improve performance on ARM processors. (If the preference is to handle PRs for both versions/branches in a single issue, feel free to close this as a duplicate.) > Update Netty 4.0.44 -> 4.1.50 (fix security/performance issues) > --- > > Key: CASSANDRA-15873 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15873 > Project: Cassandra > Issue Type: Task > Components: Dependencies >Reporter: Matt Davis >Priority: Normal > Attachments: dependency-check-report.html > > > See https://issues.apache.org/jira/browse/CASSANDRA-15868 for the same issue > on 4.0 / trunk. Attached is an OWASP dependency report for Netty 4.0.44, > which identifies 3 of the same vulnerabilities as above. > > Additionally, 4.1.50 contains aarch64 native libraries which can improve > performance on ARM processors. > > (If the preference is to handle PRs for both versions/branches in a single > issue, feel free to close this as a duplicate.) > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15873) Update Netty 4.0.44 -> 4.1.50 (fix security/performance issues)
Matt Davis created CASSANDRA-15873: -- Summary: Update Netty 4.0.44 -> 4.1.50 (fix security/performance issues) Key: CASSANDRA-15873 URL: https://issues.apache.org/jira/browse/CASSANDRA-15873 Project: Cassandra Issue Type: Task Components: Dependencies Reporter: Matt Davis Attachments: dependency-check-report.html See https://issues.apache.org/jira/browse/CASSANDRA-15868 for the same issue on 4.0 / trunk. Attached is an OWASP dependency report for Netty 4.0.44, which identifies 3 of the same vulnerabilities as the above issue. Additionally, 4.1.50 contains aarch64 native libraries which can improve performance on ARM processors. (If the preference is to handle PRs for both versions/branches in a single issue, feel free to close this as a duplicate.) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15299) CASSANDRA-13304 follow-up: improve checksumming and compression in protocol v5-beta
[
https://issues.apache.org/jira/browse/CASSANDRA-15299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhaoYang updated CASSANDRA-15299:
-
Reviewers: ZhaoYang
> CASSANDRA-13304 follow-up: improve checksumming and compression in protocol
> v5-beta
> ---
>
> Key: CASSANDRA-15299
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15299
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Client
>Reporter: Aleksey Yeschenko
>Assignee: Sam Tunnicliffe
>Priority: Normal
> Labels: protocolv5
> Fix For: 4.0-alpha
>
>
> CASSANDRA-13304 made an important improvement to our native protocol: it
> introduced checksumming/CRC32 to request and response bodies. It’s an
> important step forward, but it doesn’t cover the entire stream. In
> particular, the message header is not covered by a checksum or a crc, which
> poses a correctness issue if, for example, {{streamId}} gets corrupted.
> Additionally, we aren’t quite using CRC32 correctly, in two ways:
> 1. We are calculating the CRC32 of the *decompressed* value instead of
> computing the CRC32 on the bytes written on the wire - losing the properties
> of the CRC32. In some cases, due to this sequencing, attempting to decompress
> a corrupt stream can cause a segfault by LZ4.
> 2. When using CRC32, the CRC32 value is written in the incorrect byte order,
> also losing some of the protections.
> See https://users.ece.cmu.edu/~koopman/pubs/KoopmanCRCWebinar9May2012.pdf for
> explanation for the two points above.
> Separately, there are some long-standing issues with the protocol - since
> *way* before CASSANDRA-13304. Importantly, both checksumming and compression
> operate on individual message bodies rather than frames of multiple complete
> messages. In reality, this has several important additional downsides. To
> name a couple:
> # For compression, we are getting poor compression ratios for smaller
> messages - when operating on tiny sequences of bytes. In reality, for most
> small requests and responses we are discarding the compressed value as it’d
> be smaller than the uncompressed one - incurring both redundant allocations
> and compressions.
> # For checksumming and CRC32 we pay a high overhead price for small messages.
> 4 bytes extra is *a lot* for an empty write response, for example.
> To address the correctness issue of {{streamId}} not being covered by the
> checksum/CRC32 and the inefficiency in compression and checksumming/CRC32, we
> should switch to a framing protocol with multiple messages in a single frame.
> I suggest we reuse the framing protocol recently implemented for internode
> messaging in CASSANDRA-15066 to the extent that its logic can be borrowed,
> and that we do it before native protocol v5 graduates from beta. See
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderCrc.java
> and
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderLZ4.java.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15752) Range read concurrency factor didn't consider range merger
[
https://issues.apache.org/jira/browse/CASSANDRA-15752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134358#comment-17134358
]
Caleb Rackliffe commented on CASSANDRA-15752:
-
+1
> Range read concurrency factor didn't consider range merger
> --
>
> Key: CASSANDRA-15752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15752
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Coordination
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
>
> During range read, coordinator computes concurrency factor which is the
> number of vnode ranges to contact in parallel for the next batch.
> But in {{RangeCommandIterator}}, vnode ranges are merged by {{RangeMerger}}
> if vnode ranges share enough replicas to satisfy consistency level. eg. vnode
> range [a,b) has replica n1,n2,n3 and vnode range [b,c) has replica n2,n3,n4,
> so they can be merged as range [a,c) with replica n2, n3 for Quorum.
> Currently it counts number of merged ranges towards concurrency factor.
> Coordinator may fetch more ranges than needed.
>
> Another issue is that when executing range read on table with very small
> amount of data, concurrency factor can be bumped to {{size of total vnode
> ranges}}, eg. 10k, depending on the num of vnodes and cluster size. As a
> result, coordinator will send large number of concurrent range requests,
> potentially slowing down the cluster.. We should cap the max concurrency
> factor..
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14819) Certain table metrics not released when table is dropped.
[
https://issues.apache.org/jira/browse/CASSANDRA-14819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Caleb Rackliffe updated CASSANDRA-14819:
Resolution: Duplicate
Status: Resolved (was: Open)
CASSANDRA-14888 includes fixes for these metric releases.
> Certain table metrics not released when table is dropped.
> -
>
> Key: CASSANDRA-14819
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14819
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Adam Zegelin
>Priority: Low
> Attachments: release-metrics.patch
>
>
> The following metrics are not released when a table is dropped:
> * {{CasCommitLatency}}
> * {{CasCommitTotalLatency}}
> * {{CasPrepareLatency}}
> * {{CasPrepareTotalLatency}}
> * {{CasProposeLatency}}
> * {{CasProposeTotalLatency}}
> * {{ReadRepairRequests}}
> * {{ShortReadProtectionRequests}}
>
> This can be verified by inspecting the list of exported MBeans after creating
> a table then dropping it.
> The result is that if a table with the same name is re-created then the above
> metrics wont be updated - internally a
> {{javax.management.InstanceAlreadyExistsException}} is thrown by the
> {{MBeanServer}} when the new table's metric is registered, but this exception
> is silently ignored by
> {{org.apache.cassandra.metrics.CassandraMetricsRegistry#registerMBean}}
> The above metrics were added in 2014/2015, so this issue probably affects all
> versions released since then.
>
> Attached a patch that releases/removes the metrics when the table is dropped.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14888) Several mbeans are not unregistered when dropping a keyspace and table
[ https://issues.apache.org/jira/browse/CASSANDRA-14888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Caleb Rackliffe updated CASSANDRA-14888: Test and Documentation Plan: Review and tests CircleCI: https://app.circleci.com/pipelines/github/maedhroz/cassandra?branch=14888-maedhroz was: Review and tests CircleCI: https://app.circleci.com/pipelines/github/maedhroz/cassandra/5/workflows/357e99c0-fa27-4027-8b01-3909eac71e4c > Several mbeans are not unregistered when dropping a keyspace and table > -- > > Key: CASSANDRA-14888 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14888 > Project: Cassandra > Issue Type: Bug > Components: Observability/Metrics >Reporter: Ariel Weisberg >Assignee: Alex Deparvu >Priority: Urgent > Labels: patch-available > Fix For: 3.0.x, 3.11.x, 4.0-beta > > Attachments: CASSANDRA-14888.patch > > Time Spent: 2h 20m > Remaining Estimate: 0h > > CasCommit, CasPrepare, CasPropose, ReadRepairRequests, > ShortReadProtectionRequests, AntiCompactionTime, BytesValidated, > PartitionsValidated, RepairPrepareTime, RepairSyncTime, > RepairedDataInconsistencies, ViewLockAcquireTime, ViewReadTime, > WriteFailedIdealCL > Basically for 3 years people haven't known what they are doing because the > entire thing is kind of obscure. Fix it and also add a dtest that detects if > any mbeans are left behind after dropping a table and keyspace. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14888) Several mbeans are not unregistered when dropping a keyspace and table
[ https://issues.apache.org/jira/browse/CASSANDRA-14888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134312#comment-17134312 ] Alex Deparvu commented on CASSANDRA-14888: -- thanks [~maedhroz] let me know how the CI results look like. just in case it got missed in the wall of updates, this PR also includes CASSANDRA-14819. > Several mbeans are not unregistered when dropping a keyspace and table > -- > > Key: CASSANDRA-14888 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14888 > Project: Cassandra > Issue Type: Bug > Components: Observability/Metrics >Reporter: Ariel Weisberg >Assignee: Alex Deparvu >Priority: Urgent > Labels: patch-available > Fix For: 3.0.x, 3.11.x, 4.0-beta > > Attachments: CASSANDRA-14888.patch > > Time Spent: 2h 20m > Remaining Estimate: 0h > > CasCommit, CasPrepare, CasPropose, ReadRepairRequests, > ShortReadProtectionRequests, AntiCompactionTime, BytesValidated, > PartitionsValidated, RepairPrepareTime, RepairSyncTime, > RepairedDataInconsistencies, ViewLockAcquireTime, ViewReadTime, > WriteFailedIdealCL > Basically for 3 years people haven't known what they are doing because the > entire thing is kind of obscure. Fix it and also add a dtest that detects if > any mbeans are left behind after dropping a table and keyspace. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15861) Mutating sstable STATS metadata may race with entire-sstable-streaming(ZCS) causing checksum validation failure
[
https://issues.apache.org/jira/browse/CASSANDRA-15861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhaoYang updated CASSANDRA-15861:
-
Description:
Flaky dtest: [test_dead_sync_initiator -
repair_tests.repair_test.TestRepair|https://ci-cassandra.apache.org/view/all/job/Cassandra-devbranch-dtest/143/testReport/junit/dtest.repair_tests.repair_test/TestRepair/test_dead_sync_initiator/]
In the above test, it executes "nodetool repair" on node1 and kills node2
during repair. At the end, node3 reports checksum validation failure on sstable
transferred from node1.
{code:java|title=what happened}
1. When repair started on node1, it performs anti-compaction which modifies
sstable's repairAt to 0 and pending repair id to session-id.
2. Then node1 creates {{ComponentManifest}} which contains file lengths to be
transferred to node3.
3. Before node1 actually sends the files to node3, node2 is killed and node1
starts to broadcast repair-failure-message to all participants in
{{CoordinatorSession#fail}}
4. Node1 receives its own repair-failure-message and fails its local repair
sessions at {{LocalSessions#failSession}} which triggers async background
compaction.
5. Node1's background compaction will mutate sstable's repairAt to 0 and
pending repair id to null via
{{PendingRepairManager#getNextRepairFinishedTask}}, as there is no more
in-progress repair.
6. Node1 actually sends the sstable to node3 where the sstable's STATS
component size is different from the original size recorded in the manifest.
7. At the end, node3 reports checksum validation failure when it tries to
mutate sstable level and "isTransient" attribute in
{{CassandraEntireSSTableStreamReader#read}}.
{code}
I believe similar race may happen with level compaction where it may directly
mutate a sstable's level if it doesn't overlap with sstables at next level.
(Note: this isn't a problem in legacy streaming as STATS file length didn't
matter.)
Ideally it will be great to make sstable STATS metadata immutable, just like
other sstable components, so we don't have to worry this special case.
I can think of two ways:
# Change {{RepairFinishedCompactionTask}} and {{SingleSSTableLCSTask}} to
create hard link on the compacting sstable components with a new descriptor,
except STATS files which will be copied entirely. Then mutation will be applied
on the new STATS file. At the end, old sstable will be released. This ensures
all sstable components are immutable and shouldn't make these special
compaction tasks slower.
# Hacky approach: load the small STATS file into memory when initializing
{{CassandraOutgoingFile}} instead of relying on mutable on-disk STATS file.
was:
Flaky dtest: [test_dead_sync_initiator -
repair_tests.repair_test.TestRepair|https://ci-cassandra.apache.org/view/all/job/Cassandra-devbranch-dtest/143/testReport/junit/dtest.repair_tests.repair_test/TestRepair/test_dead_sync_initiator/]
In the above test, it executes "nodetool repair" on node1 and kills node2
during repair. At the end, node3 reports checksum validation failure on sstable
transferred from node1.
{code:java|title=what happened}
1. When repair started on node1, it performs anti-compaction which modifies
sstable's repairAt to 0 and pending repair id to session-id.
2. Then node1 creates {{ComponentManifest}} which contains file lengths to be
transferred to node3.
3. Before node1 actually sends the files to node3, node2 is killed and node1
starts to broadcast repair-failure-message to all participants in
{{CoordinatorSession#fail}}
4. Node1 receives its own repair-failure-message and fails its local repair
sessions at {{LocalSessions#failSession}} which triggers async background
compaction.
5. Node1's background compaction will mutate sstable's repairAt to 0 and
pending repair id to null via
{{PendingRepairManager#getNextRepairFinishedTask}}, as there is no more
in-progress repair.
6. Node1 actually sends the sstable to node3 where the sstable's STATS
component size is different from the original size recorded in the manifest.
7. At the end, node3 reports checksum validation failure when it tries to
mutate sstable level and "isTransient" attribute in
{{CassandraEntireSSTableStreamReader#read}}.
{code}
I believe similar race may happen with level compaction where it may directly
mutate a sstable's level if it doesn't overlap with sstables at next level.
(Note: this isn't a problem in legacy streaming as STATS file length didn't
matter.)
Ideally it will be great to make sstable STATS metadata immutable, just like
other sstable components, so we don't have to worry this special case.
I can think of two ways:
# Change {{RepairFinishedCompactionTask}} and {{SingleSSTableLCSTask}} to
create hard link on the compacting sstable components with a new descriptor,
except STATS files which will be copied entirely. Then mutation will be applied
on the new STATS file.
[jira] [Updated] (CASSANDRA-15861) Mutating sstable STATS metadata may race with entire-sstable-streaming(ZCS) causing checksum validation failure
[
https://issues.apache.org/jira/browse/CASSANDRA-15861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhaoYang updated CASSANDRA-15861:
-
Description:
Flaky dtest: [test_dead_sync_initiator -
repair_tests.repair_test.TestRepair|https://ci-cassandra.apache.org/view/all/job/Cassandra-devbranch-dtest/143/testReport/junit/dtest.repair_tests.repair_test/TestRepair/test_dead_sync_initiator/]
In the above test, it executes "nodetool repair" on node1 and kills node2
during repair. At the end, node3 reports checksum validation failure on sstable
transferred from node1.
{code:java|title=what happened}
1. When repair started on node1, it performs anti-compaction which modifies
sstable's repairAt to 0 and pending repair id to session-id.
2. Then node1 creates {{ComponentManifest}} which contains file lengths to be
transferred to node3.
3. Before node1 actually sends the files to node3, node2 is killed and node1
starts to broadcast repair-failure-message to all participants in
{{CoordinatorSession#fail}}
4. Node1 receives its own repair-failure-message and fails its local repair
sessions at {{LocalSessions#failSession}} which triggers async background
compaction.
5. Node1's background compaction will mutate sstable's repairAt to 0 and
pending repair id to null via
{{PendingRepairManager#getNextRepairFinishedTask}}, as there is no more
in-progress repair.
6. Node1 actually sends the sstable to node3 where the sstable's STATS
component size is different from the original size recorded in the manifest.
7. At the end, node3 reports checksum validation failure when it tries to
mutate sstable level and "isTransient" attribute in
{{CassandraEntireSSTableStreamReader#read}}.
{code}
I believe similar race may happen with level compaction where it may directly
mutate a sstable's level if it doesn't overlap with sstables at next level.
(Note: this isn't a problem in legacy streaming as STATS file length didn't
matter.)
Ideally it will be great to make sstable STATS metadata immutable, just like
other sstable components, so we don't have to worry this special case.
I can think of two ways:
# Change {{RepairFinishedCompactionTask}} and {{SingleSSTableLCSTask}} to
create hard link on the compacting sstable components with a new descriptor,
except STATS files which will be copied entirely. Then mutation will be applied
on the new STATS file. At the end, old sstable will be released. This ensures
all sstable components are immutable.
# Hacky approach: load the small STATS file into memory when initializing
{{CassandraOutgoingFile}} instead of relying on mutable on-disk STATS file.
was:
Flaky dtest: [test_dead_sync_initiator -
repair_tests.repair_test.TestRepair|https://ci-cassandra.apache.org/view/all/job/Cassandra-devbranch-dtest/143/testReport/junit/dtest.repair_tests.repair_test/TestRepair/test_dead_sync_initiator/]
In the above test, it executes "nodetool repair" on node1 and kills node2
during repair. At the end, node3 reports checksum validation failure on sstable
transferred from node1.
{code:java|title=what happened}
1. When repair started on node1, it performs anti-compaction which modifies
sstable's repairAt to 0 and pending repair id to session-id.
2. Then node1 creates {{ComponentManifest}} which contains file lengths to be
transferred to node3.
3. Before node1 actually sends the files to node3, node2 is killed and node1
starts to broadcast repair-failure-message to all participants in
{{CoordinatorSession#fail}}
4. Node1 receives its own repair-failure-message and fails its local repair
sessions at {{LocalSessions#failSession}} which triggers async background
compaction.
5. Node1's background compaction will mutate sstable's repairAt to 0 and
pending repair id to null via
{{PendingRepairManager#getNextRepairFinishedTask}}, as there is no more
in-progress repair.
6. Node1 actually sends the sstable to node3 where the sstable's STATS
component size is different from the original size recorded in the manifest.
7. At the end, node3 reports checksum validation failure when it tries to
mutate sstable level and "isTransient" attribute in
{{CassandraEntireSSTableStreamReader#read}}.
{code}
I believe similar race may happen with level compaction where it may directly
mutate a sstable's level if it doesn't overlap with sstables at next level.
(Note: this isn't a problem in legacy streaming as STATS file length didn't
matter.)
Ideally it will be great to make sstable STATS metadata immutable, just like
other sstable components, so we don't have to worry this special case.
I can think of two ways:
1. Change {{RepairFinishedCompactionTask}} and {{SingleSSTableLCSTask}} to
create hard link on the compacting sstable components with a new descriptor,
except STATS files which will be copied entirely. Then mutation will be applied
on the new STATS file. At the end, old sstable will be released. This ensures
[jira] [Updated] (CASSANDRA-15861) Mutating sstable STATS metadata may race with entire-sstable-streaming(ZCS) causing checksum validation failure
[
https://issues.apache.org/jira/browse/CASSANDRA-15861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhaoYang updated CASSANDRA-15861:
-
Description:
Flaky dtest: [test_dead_sync_initiator -
repair_tests.repair_test.TestRepair|https://ci-cassandra.apache.org/view/all/job/Cassandra-devbranch-dtest/143/testReport/junit/dtest.repair_tests.repair_test/TestRepair/test_dead_sync_initiator/]
In the above test, it executes "nodetool repair" on node1 and kills node2
during repair. At the end, node3 reports checksum validation failure on sstable
transferred from node1.
{code:java|title=what happened}
1. When repair started on node1, it performs anti-compaction which modifies
sstable's repairAt to 0 and pending repair id to session-id.
2. Then node1 creates {{ComponentManifest}} which contains file lengths to be
transferred to node3.
3. Before node1 actually sends the files to node3, node2 is killed and node1
starts to broadcast repair-failure-message to all participants in
{{CoordinatorSession#fail}}
4. Node1 receives its own repair-failure-message and fails its local repair
sessions at {{LocalSessions#failSession}} which triggers async background
compaction.
5. Node1's background compaction will mutate sstable's repairAt to 0 and
pending repair id to null via
{{PendingRepairManager#getNextRepairFinishedTask}}, as there is no more
in-progress repair.
6. Node1 actually sends the sstable to node3 where the sstable's STATS
component size is different from the original size recorded in the manifest.
7. At the end, node3 reports checksum validation failure when it tries to
mutate sstable level and "isTransient" attribute in
{{CassandraEntireSSTableStreamReader#read}}.
{code}
I believe similar race may happen with level compaction where it may directly
mutate a sstable's level if it doesn't overlap with sstables at next level.
(Note: this isn't a problem in legacy streaming as STATS file length didn't
matter.)
Ideally it will be great to make sstable STATS metadata immutable, just like
other sstable components, so we don't have to worry this special case.
I can think of two ways:
1. Change {{RepairFinishedCompactionTask}} and {{SingleSSTableLCSTask}} to
create hard link on the compacting sstable components with a new descriptor,
except STATS files which will be copied entirely. Then mutation will be applied
on the new STATS file. At the end, old sstable will be released. This ensures
all sstable components are immutable.
2. Hacky approach: to load the small STATS file into memory when initializing
{{CassandraOutgoingFile}} instead of relying on mutable on-disk STATS file.
was:
Flaky dtest: [test_dead_sync_initiator -
repair_tests.repair_test.TestRepair|https://ci-cassandra.apache.org/view/all/job/Cassandra-devbranch-dtest/143/testReport/junit/dtest.repair_tests.repair_test/TestRepair/test_dead_sync_initiator/]
In the above test, it executes "nodetool repair" on node1 and kills node2
during repair. At the end, node3 reports checksum validation failure on sstable
transferred from node1.
{code:java|title=what happened}
1. When repair started on node1, it performs anti-compaction which modifies
sstable's repairAt to 0 and pending repair id to session-id.
2. Then node1 creates {{ComponentManifest}} which contains file lengths to be
transferred to node3.
3. Before node1 actually sends the files to node3, node2 is killed and node1
starts to broadcast repair-failure-message to all participants in
{{CoordinatorSession#fail}}
4. Node1 receives its own repair-failure-message and fails its local repair
sessions at {{LocalSessions#failSession}} which triggers async background
compaction.
5. Node1's background compaction will mutate sstable's repairAt to 0 and
pending repair id to null via
{{PendingRepairManager#getNextRepairFinishedTask}}, as there is no more
in-progress repair.
6. Node1 actually sends the sstable to node3 where the sstable's STATS
component size is different from the original size recorded in the manifest.
7. At the end, node3 reports checksum validation failure when it tries to
mutate sstable level and "isTransient" attribute in
{{CassandraEntireSSTableStreamReader#read}}.
{code}
I believe similar race may happen with level compaction where it may directly
mutate a sstable's level if it doesn't overlap with sstables at next level.
(Note: this isn't a problem in legacy streaming as STATS file length didn't
matter.)
Ideally it will be great to make sstable STATS metadata immutable, just like
other sstable components, so we don't have to worry this special case. For now,
I suggest to use a {{StatsMetadata}} snapshot when initializing
{{CassandraOutgoingFile}} instead of relying on mutable on-disk STATS file.
> Mutating sstable STATS metadata may race with entire-sstable-streaming(ZCS)
> causing checksum validation failure
>
[jira] [Updated] (CASSANDRA-14227) Extend maximum expiration date
[
https://issues.apache.org/jira/browse/CASSANDRA-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Shuler updated CASSANDRA-14227:
---
Complexity: Challenging
Severity: Critical (was: Normal)
Set this Jira to Severity = Critical and Complexity = Challenging to reflect
the state of this problem, as time goes on, as well as the nature of the fix.
> Extend maximum expiration date
> --
>
> Key: CASSANDRA-14227
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14227
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: Paulo Motta
>Priority: Urgent
>
> The maximum expiration timestamp that can be represented by the storage
> engine is
> 2038-01-19T03:14:06+00:00 due to the encoding of {{localExpirationTime}} as
> an int32.
> On CASSANDRA-14092 we added an overflow policy which rejects requests with
> expiration above the maximum date as a temporary measure, but we should
> remove this limitation by updating the storage engine to support at least the
> maximum allowed TTL of 20 years.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15833) Unresolvable false digest mismatch during upgrade due to CASSANDRA-10657
[ https://issues.apache.org/jira/browse/CASSANDRA-15833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134297#comment-17134297 ] Jacek Lewandowski commented on CASSANDRA-15833: --- Yup, I'll finish that, just was a bit busy recently > Unresolvable false digest mismatch during upgrade due to CASSANDRA-10657 > > > Key: CASSANDRA-15833 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15833 > Project: Cassandra > Issue Type: Bug > Components: Consistency/Repair >Reporter: Jacek Lewandowski >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 3.11.x, 4.0-alpha > > Attachments: CASSANDRA-15833-3.11.patch, CASSANDRA-15833-4.0.patch > > > CASSANDRA-10657 introduced changes in how the ColumnFilter is interpreted. > This results in digest mismatch when querying incomplete set of columns from > a table with consistency that requires reaching instances running pre > CASSANDRA-10657 from nodes that include CASSANDRA-10657 (it was introduced in > Cassandra 3.4). > The fix is to bring back the previous behaviour until there are no instances > running pre CASSANDRA-10657 version. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15869) MemoryOutputStream overflow on large bloom filter
[
https://issues.apache.org/jira/browse/CASSANDRA-15869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134296#comment-17134296
]
ZhaoYang commented on CASSANDRA-15869:
--
thank you
> MemoryOutputStream overflow on large bloom filter
> -
>
> Key: CASSANDRA-15869
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15869
> Project: Cassandra
> Issue Type: Bug
> Components: Local/SSTable
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 4.0-alpha5
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Since CASSANDRA-9067, Cassandra will use {{MemoryOutputStream}} to
> reconstruct BF Memory without re-ordering bytes, see
> {{OffsetBitSet#deserialize}}. But {{MemoryOutputStream}} use {{INT}} to track
> position and will overflow when BF size exceeds 2GB.
> {code:java|title=stacktrace}
> error: Illegal bounds [-2147483648..-2147483584); size: 4588588016
> -- StackTrace --
> java.lang.AssertionError: Illegal bounds [-2147483648..-2147483584); size:
> 4588588016
> at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:185)
> at org.apache.cassandra.io.util.Memory.setBytes(Memory.java:138)
> at
> org.apache.cassandra.io.util.MemoryOutputStream.write(MemoryOutputStream.java:45)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15872) Flaky test: testNoTreesRetainedAfterDifference - org.apache.cassandra.repair.RepairJobTest
[ https://issues.apache.org/jira/browse/CASSANDRA-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134293#comment-17134293 ] ZhaoYang commented on CASSANDRA-15872: -- [patch|https://github.com/apache/cassandra/pull/630/files]: the test is flaky when repair task executor executes the job before the assertion. the fix is to put a latch. > Flaky test: testNoTreesRetainedAfterDifference - > org.apache.cassandra.repair.RepairJobTest > -- > > Key: CASSANDRA-15872 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15872 > Project: Cassandra > Issue Type: Task > Components: Consistency/Repair >Reporter: ZhaoYang >Priority: Normal > Fix For: 4.0-rc > > > link: [https://circleci.com/gh/maedhroz/cassandra/21#tests/containers/95] > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15872) Flaky test: testNoTreesRetainedAfterDifference - org.apache.cassandra.repair.RepairJobTest
[ https://issues.apache.org/jira/browse/CASSANDRA-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134293#comment-17134293 ] ZhaoYang edited comment on CASSANDRA-15872 at 6/12/20, 2:58 PM: [patch|https://github.com/apache/cassandra/pull/630/files]: the test is flaky when repair task executor executes the job before the assertion. the fix is to put a latch. cc [~maedhroz] was (Author: jasonstack): [patch|https://github.com/apache/cassandra/pull/630/files]: the test is flaky when repair task executor executes the job before the assertion. the fix is to put a latch. > Flaky test: testNoTreesRetainedAfterDifference - > org.apache.cassandra.repair.RepairJobTest > -- > > Key: CASSANDRA-15872 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15872 > Project: Cassandra > Issue Type: Task > Components: Consistency/Repair >Reporter: ZhaoYang >Priority: Normal > Fix For: 4.0-rc > > > link: [https://circleci.com/gh/maedhroz/cassandra/21#tests/containers/95] > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15872) Flaky test: testNoTreesRetainedAfterDifference - org.apache.cassandra.repair.RepairJobTest
[ https://issues.apache.org/jira/browse/CASSANDRA-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhaoYang updated CASSANDRA-15872: - Fix Version/s: 4.0-rc > Flaky test: testNoTreesRetainedAfterDifference - > org.apache.cassandra.repair.RepairJobTest > -- > > Key: CASSANDRA-15872 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15872 > Project: Cassandra > Issue Type: Task > Components: Consistency/Repair >Reporter: ZhaoYang >Priority: Normal > Fix For: 4.0-rc > > > link: [https://circleci.com/gh/maedhroz/cassandra/21#tests/containers/95] > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15872) Flaky test: testNoTreesRetainedAfterDifference - org.apache.cassandra.repair.RepairJobTest
ZhaoYang created CASSANDRA-15872: Summary: Flaky test: testNoTreesRetainedAfterDifference - org.apache.cassandra.repair.RepairJobTest Key: CASSANDRA-15872 URL: https://issues.apache.org/jira/browse/CASSANDRA-15872 Project: Cassandra Issue Type: Task Components: Consistency/Repair Reporter: ZhaoYang link: [https://circleci.com/gh/maedhroz/cassandra/21#tests/containers/95] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15871) Cassandra driver first connection get the user's own schema information
[ https://issues.apache.org/jira/browse/CASSANDRA-15871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-15871: - Resolution: Invalid Status: Resolved (was: Triage Needed) You need to open a ticket against whichever driver you are using. > Cassandra driver first connection get the user's own schema information > > > Key: CASSANDRA-15871 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15871 > Project: Cassandra > Issue Type: Improvement > Components: Cluster/Schema, Messaging/Client >Reporter: maxwellguo >Priority: Normal > > We know that cassandra driver making a conenction with the coordinator node > first time , the driver may select all the keyspaces/tables/columns/types > from the server and cache the data locally. > For different users they may have different tables and types ，so It is not > suitable to get all the meta data cached , It is fine to just cache the > user's own schema information not all. > And doing this is safe and save first time connection resourse. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15863) Bootstrap resume and TestReplaceAddress fixes
[ https://issues.apache.org/jira/browse/CASSANDRA-15863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-15863: - Reviewers: Brandon Williams, Brandon Williams (was: Brandon Williams) Brandon Williams, Brandon Williams (was: Brandon Williams) Status: Review In Progress (was: Patch Available) > Bootstrap resume and TestReplaceAddress fixes > - > > Key: CASSANDRA-15863 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15863 > Project: Cassandra > Issue Type: Bug > Components: Consistency/Bootstrap and Decommission, Test/dtest >Reporter: Berenguer Blasi >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.0-alpha > > Time Spent: 10m > Remaining Estimate: 0h > > This has been > [broken|https://ci-cassandra.apache.org/job/Cassandra-trunk/159/testReport/dtest-large.replace_address_test/TestReplaceAddress/test_restart_failed_replace/history/] > for ages -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15834) Bloom filter false positive rate calculation does not take into account true negatives
[
https://issues.apache.org/jira/browse/CASSANDRA-15834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-15834:
-
Reviewers: Brandon Williams
> Bloom filter false positive rate calculation does not take into account true
> negatives
> --
>
> Key: CASSANDRA-15834
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15834
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/JMX
>Reporter: Jaroslaw Grabowski
>Assignee: Jaroslaw Grabowski
>Priority: Normal
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The bloom filter false positive ratio is [currently
> computed|https://github.com/apache/cassandra/blob/ded62076e7fdfd1cfdcf96447489ea607ca796a0/src/java/org/apache/cassandra/metrics/TableMetrics.java#L738]
> as:
> {{bf_fp_ratio = false_positive_count / (false_positive_count +
> true_positive_count)}}
> However, this calculation doesn't take into account true negatives (false
> negatives never happen on bloom filters).
> In a situation where there are 1000 reads for non existing rows, and there
> are 10 false positives, the bloom filter false positive ratio will be wrongly
> calculated as 10/10 = 1.0, while it should be 10/1000 = 0.01.
> We should update the calculation to:
> {{bf_fp_ratio = false_positive_count / #bf_queries}}
> Original jira by [~pauloricardomg]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15869) MemoryOutputStream overflow on large bloom filter
[
https://issues.apache.org/jira/browse/CASSANDRA-15869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-15869:
-
Fix Version/s: (was: 4.0-beta)
4.0-alpha5
Since Version: 4.0-alpha1
Source Control Link:
https://github.com/apache/cassandra/commit/7cdad3ce61664cf2ffac75b6e537c910e573754c
(was: https://github.com/apache/cassandra/pull/620)
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
Thanks, committed.
> MemoryOutputStream overflow on large bloom filter
> -
>
> Key: CASSANDRA-15869
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15869
> Project: Cassandra
> Issue Type: Bug
> Components: Local/SSTable
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 4.0-alpha5
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Since CASSANDRA-9067, Cassandra will use {{MemoryOutputStream}} to
> reconstruct BF Memory without re-ordering bytes, see
> {{OffsetBitSet#deserialize}}. But {{MemoryOutputStream}} use {{INT}} to track
> position and will overflow when BF size exceeds 2GB.
> {code:java|title=stacktrace}
> error: Illegal bounds [-2147483648..-2147483584); size: 4588588016
> -- StackTrace --
> java.lang.AssertionError: Illegal bounds [-2147483648..-2147483584); size:
> 4588588016
> at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:185)
> at org.apache.cassandra.io.util.Memory.setBytes(Memory.java:138)
> at
> org.apache.cassandra.io.util.MemoryOutputStream.write(MemoryOutputStream.java:45)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15869) MemoryOutputStream overflow on large bloom filter
[
https://issues.apache.org/jira/browse/CASSANDRA-15869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-15869:
-
Reviewers: Brandon Williams, Brandon Williams (was: Brandon Williams)
Brandon Williams, Brandon Williams
Status: Review In Progress (was: Patch Available)
> MemoryOutputStream overflow on large bloom filter
> -
>
> Key: CASSANDRA-15869
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15869
> Project: Cassandra
> Issue Type: Bug
> Components: Local/SSTable
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 4.0-beta
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Since CASSANDRA-9067, Cassandra will use {{MemoryOutputStream}} to
> reconstruct BF Memory without re-ordering bytes, see
> {{OffsetBitSet#deserialize}}. But {{MemoryOutputStream}} use {{INT}} to track
> position and will overflow when BF size exceeds 2GB.
> {code:java|title=stacktrace}
> error: Illegal bounds [-2147483648..-2147483584); size: 4588588016
> -- StackTrace --
> java.lang.AssertionError: Illegal bounds [-2147483648..-2147483584); size:
> 4588588016
> at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:185)
> at org.apache.cassandra.io.util.Memory.setBytes(Memory.java:138)
> at
> org.apache.cassandra.io.util.MemoryOutputStream.write(MemoryOutputStream.java:45)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15869) MemoryOutputStream overflow on large bloom filter
[
https://issues.apache.org/jira/browse/CASSANDRA-15869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-15869:
-
Status: Ready to Commit (was: Review In Progress)
> MemoryOutputStream overflow on large bloom filter
> -
>
> Key: CASSANDRA-15869
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15869
> Project: Cassandra
> Issue Type: Bug
> Components: Local/SSTable
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 4.0-beta
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Since CASSANDRA-9067, Cassandra will use {{MemoryOutputStream}} to
> reconstruct BF Memory without re-ordering bytes, see
> {{OffsetBitSet#deserialize}}. But {{MemoryOutputStream}} use {{INT}} to track
> position and will overflow when BF size exceeds 2GB.
> {code:java|title=stacktrace}
> error: Illegal bounds [-2147483648..-2147483584); size: 4588588016
> -- StackTrace --
> java.lang.AssertionError: Illegal bounds [-2147483648..-2147483584); size:
> 4588588016
> at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:185)
> at org.apache.cassandra.io.util.Memory.setBytes(Memory.java:138)
> at
> org.apache.cassandra.io.util.MemoryOutputStream.write(MemoryOutputStream.java:45)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated: Avoid overflow when bloom filter size exceeds 2GB
This is an automated email from the ASF dual-hosted git repository.
brandonwilliams pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new 7cdad3c Avoid overflow when bloom filter size exceeds 2GB
7cdad3c is described below
commit 7cdad3ce61664cf2ffac75b6e537c910e573754c
Author: Zhao Yang
AuthorDate: Thu Jun 11 13:05:47 2020 +0800
Avoid overflow when bloom filter size exceeds 2GB
Patch by Zhao Yang, reviewed by brandonwilliams for CASSANDRA-15869
---
src/java/org/apache/cassandra/io/util/MemoryOutputStream.java | 7 +--
1 file changed, 1 insertion(+), 6 deletions(-)
diff --git a/src/java/org/apache/cassandra/io/util/MemoryOutputStream.java
b/src/java/org/apache/cassandra/io/util/MemoryOutputStream.java
index e6c869d..c984738 100644
--- a/src/java/org/apache/cassandra/io/util/MemoryOutputStream.java
+++ b/src/java/org/apache/cassandra/io/util/MemoryOutputStream.java
@@ -27,7 +27,7 @@ public class MemoryOutputStream extends OutputStream
{
private final Memory mem;
-private int position = 0;
+private long position = 0;
public MemoryOutputStream(Memory mem)
{
@@ -45,9 +45,4 @@ public class MemoryOutputStream extends OutputStream
mem.setBytes(position, b, off, len);
position += len;
}
-
-public int position()
-{
-return position;
-}
}
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15867) Update Jackson version to 2.9.10.1 because there are security issues in 2.9.5
[ https://issues.apache.org/jira/browse/CASSANDRA-15867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134251#comment-17134251 ] Brandon Williams commented on CASSANDRA-15867: -- We can merge it into 3.11 as well, your patch was against trunk so I just went with that. The patch doesn't cleanly apply to 3.11. > Update Jackson version to 2.9.10.1 because there are security issues in 2.9.5 > - > > Key: CASSANDRA-15867 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15867 > Project: Cassandra > Issue Type: Task > Components: Dependencies >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0-alpha5 > > Attachments: dependency-check-report.html > > > Please see attached HTML report from OWASP dependency check for current > 4.0-alpha5 trunk branch. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12126) CAS Reads Inconsistencies
[
https://issues.apache.org/jira/browse/CASSANDRA-12126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134239#comment-17134239
]
Sylvain Lebresne commented on CASSANDRA-12126:
--
Ok, I've rebased the patch against 4.0 and started CI on it all:
||branch||CI||
|[3.0|https://github.com/pcmanus/cassandra/tree/C-12126-3.0]|[Run
#146|https://ci-cassandra.apache.org/job/Cassandra-devbranch/146/]|
|[3.11|https://github.com/pcmanus/cassandra/tree/C-12126-3.11]|[Run
#147|https://ci-cassandra.apache.org/job/Cassandra-devbranch/147/]|
|[4.0|https://github.com/pcmanus/cassandra/tree/C-12126-4.0]|[Run
#148|https://ci-cassandra.apache.org/job/Cassandra-devbranch/148/]|
I included a commit to add the flag that disables the new empty commit for
SERIAL reads as suggested by [~bdeggleston] earlier. Still slightly on the
fence on the need for such flag, but I call it "unsafe"
({{-Dcassandra.unsafe.disable-serial-reads-linearizability}} to be specific)
and log a warning when used, so I'm at peace with that.
I'll note for future reviewers that while the 3.11 branch is almost a straight
away merge up of 3.0, there is a minor differences on the 4.0 branch, namely:
* the added in-jvm dtests needed a few changes to reflect 4.0 changes. To make
that easier, I squashed 2 of the commits from the 3.0/3.11 branches, which is
why that branch has one less commit.
* There is a few changes related to the translation of
{{WriteTimeoutException}} into {{CasWriteTimeoutException}} (I pushed it down
in some cases). I believe this fixes a minor "bug" where the "contentions"
number we returned with {{CasWriteTimeoutException}} was potentially inaccurate
(namely, if we timed out in {{beginRepairAndPaxos}}, contention leading to that
exception would be ignored)
I'll wait on getting usable CI results to officially mark it 'ready to review',
but it is in spirit if anyone is burning to look at this.
> CAS Reads Inconsistencies
> --
>
> Key: CASSANDRA-12126
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12126
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Lightweight Transactions, Legacy/Coordination
>Reporter: Sankalp Kohli
>Assignee: Sylvain Lebresne
>Priority: Normal
> Labels: LWT, pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> While looking at the CAS code in Cassandra, I found a potential issue with
> CAS Reads. Here is how it can happen with RF=3
> 1) You issue a CAS Write and it fails in the propose phase. A machine replies
> true to a propose and saves the commit in accepted filed. The other two
> machines B and C does not get to the accept phase.
> Current state is that machine A has this commit in paxos table as accepted
> but not committed and B and C does not.
> 2) Issue a CAS Read and it goes to only B and C. You wont be able to read the
> value written in step 1. This step is as if nothing is inflight.
> 3) Issue another CAS Read and it goes to A and B. Now we will discover that
> there is something inflight from A and will propose and commit it with the
> current ballot. Now we can read the value written in step 1 as part of this
> CAS read.
> If we skip step 3 and instead run step 4, we will never learn about value
> written in step 1.
> 4. Issue a CAS Write and it involves only B and C. This will succeed and
> commit a different value than step 1. Step 1 value will never be seen again
> and was never seen before.
> If you read the Lamport “paxos made simple” paper and read section 2.3. It
> talks about this issue which is how learners can find out if majority of the
> acceptors have accepted the proposal.
> In step 3, it is correct that we propose the value again since we dont know
> if it was accepted by majority of acceptors. When we ask majority of
> acceptors, and more than one acceptors but not majority has something in
> flight, we have no way of knowing if it is accepted by majority of acceptors.
> So this behavior is correct.
> However we need to fix step 2, since it caused reads to not be linearizable
> with respect to writes and other reads. In this case, we know that majority
> of acceptors have no inflight commit which means we have majority that
> nothing was accepted by majority. I think we should run a propose step here
> with empty commit and that will cause write written in step 1 to not be
> visible ever after.
> With this fix, we will either see data written in step 1 on next serial read
> or will never see it which is what we want.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15752) Range read concurrency factor didn't consider range merger
[
https://issues.apache.org/jira/browse/CASSANDRA-15752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134124#comment-17134124
]
ZhaoYang commented on CASSANDRA-15752:
--
updated the branch and rebased on trunk
> Range read concurrency factor didn't consider range merger
> --
>
> Key: CASSANDRA-15752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15752
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Coordination
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
>
> During range read, coordinator computes concurrency factor which is the
> number of vnode ranges to contact in parallel for the next batch.
> But in {{RangeCommandIterator}}, vnode ranges are merged by {{RangeMerger}}
> if vnode ranges share enough replicas to satisfy consistency level. eg. vnode
> range [a,b) has replica n1,n2,n3 and vnode range [b,c) has replica n2,n3,n4,
> so they can be merged as range [a,c) with replica n2, n3 for Quorum.
> Currently it counts number of merged ranges towards concurrency factor.
> Coordinator may fetch more ranges than needed.
>
> Another issue is that when executing range read on table with very small
> amount of data, concurrency factor can be bumped to {{size of total vnode
> ranges}}, eg. 10k, depending on the num of vnodes and cluster size. As a
> result, coordinator will send large number of concurrent range requests,
> potentially slowing down the cluster.. We should cap the max concurrency
> factor..
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15862) Use "allow list" or "safe list" instead of the term "whitelist"
[
https://issues.apache.org/jira/browse/CASSANDRA-15862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe updated CASSANDRA-15862:
Fix Version/s: (was: 4.0-alpha)
(was: 3.11.x)
(was: 3.0.x)
(was: 2.2.x)
4.0-alpha5
3.11.7
3.0.21
2.2.17
Source Control Link:
https://github.com/apache/cassandra/commit/c8c3c269bf1df944b73e2d4bf0008a6f412bf121
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
Thanks, committed to 2.2 in {{c8c3c269bf1df944b73e2d4bf0008a6f412bf121}} and
merged to trunk. I've added a deprecation notice to {{NEWS.txt}} in trunk only.
> Use "allow list" or "safe list" instead of the term "whitelist"
>
>
> Key: CASSANDRA-15862
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15862
> Project: Cassandra
> Issue Type: New Feature
> Components: Legacy/Core
>Reporter: Ash Berlin-Taylor
>Assignee: Sam Tunnicliffe
>Priority: Normal
> Fix For: 2.2.17, 3.0.21, 3.11.7, 4.0-alpha5
>
>
> Language matters. I'd like to remove all references in Apache Airflow to
> whitelist or black list, and the Cassandra Python API has some that we can't
> easily remove.
> The recent global events have made this even more relevant, but this has been
> on my radar for a while now. Here is a well written article for why I think
> it matters
> https://www.ncsc.gov.uk/blog-post/terminology-its-not-black-and-white
> {quote}It's fairly common to say whitelisting and blacklisting to describe
> desirable and undesirable things in cyber security.
> However, there's an issue with the terminology. It only makes sense if you
> equate white with 'good, permitted, safe' and black with 'bad, dangerous,
> forbidden'. There are some obvious problems with this. {quote}
> My exposure to is via the Python API where there is the
> cassandra.pollicies.WhiteListRoundRobinPolicy class. I propose that this be
> renamed to AllowListRoundRobinPolicy instead. I do not know if there are
> other references.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15862) Use "allow list" or "safe list" instead of the term "whitelist"
[
https://issues.apache.org/jira/browse/CASSANDRA-15862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe updated CASSANDRA-15862:
Status: Ready to Commit (was: Review In Progress)
> Use "allow list" or "safe list" instead of the term "whitelist"
>
>
> Key: CASSANDRA-15862
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15862
> Project: Cassandra
> Issue Type: New Feature
> Components: Legacy/Core
>Reporter: Ash Berlin-Taylor
>Assignee: Sam Tunnicliffe
>Priority: Normal
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.0-alpha
>
>
> Language matters. I'd like to remove all references in Apache Airflow to
> whitelist or black list, and the Cassandra Python API has some that we can't
> easily remove.
> The recent global events have made this even more relevant, but this has been
> on my radar for a while now. Here is a well written article for why I think
> it matters
> https://www.ncsc.gov.uk/blog-post/terminology-its-not-black-and-white
> {quote}It's fairly common to say whitelisting and blacklisting to describe
> desirable and undesirable things in cyber security.
> However, there's an issue with the terminology. It only makes sense if you
> equate white with 'good, permitted, safe' and black with 'bad, dangerous,
> forbidden'. There are some obvious problems with this. {quote}
> My exposure to is via the Python API where there is the
> cassandra.pollicies.WhiteListRoundRobinPolicy class. I propose that this be
> renamed to AllowListRoundRobinPolicy instead. I do not know if there are
> other references.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-3.0' into cassandra-3.11
This is an automated email from the ASF dual-hosted git repository. samt pushed a commit to branch cassandra-3.11 in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 408f969f3c1f6cccdb26f4ef8daec010b87933b5 Merge: e498539 0a1e8d1 Author: Sam Tunnicliffe AuthorDate: Fri Jun 12 11:26:04 2020 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 CHANGES.txt| 1 + .../cassandra/auth/jmx/AuthorizationProxy.java | 47 +++--- .../cassandra/cql3/functions/UDFunction.java | 22 +- src/java/org/apache/cassandra/db/Directories.java | 41 +-- ...Directories.java => DisallowedDirectories.java} | 29 ++--- ...sMBean.java => DisallowedDirectoriesMBean.java} | 2 +- .../org/apache/cassandra/db/DiskBoundaries.java| 2 +- .../apache/cassandra/db/DiskBoundaryManager.java | 6 +-- .../db/compaction/AbstractCompactionStrategy.java | 6 +-- .../cassandra/db/compaction/LeveledManifest.java | 2 +- .../org/apache/cassandra/dht/RangeStreamer.java| 10 ++--- .../cassandra/hints/HintsDispatchExecutor.java | 2 +- .../org/apache/cassandra/hints/HintsStore.java | 10 ++--- .../cassandra/io/sstable/format/SSTableReader.java | 2 +- .../cassandra/service/DefaultFSErrorHandler.java | 6 +-- .../apache/cassandra/service/StorageService.java | 2 +- test/unit/org/apache/cassandra/Util.java | 4 +- .../cassandra/auth/jmx/AuthorizationProxyTest.java | 4 +- .../org/apache/cassandra/db/DirectoriesTest.java | 2 +- .../cassandra/db/DiskBoundaryManagerTest.java | 6 +-- java => CorruptedSSTablesCompactionsTest.java} | 17 21 files changed, 112 insertions(+), 111 deletions(-) diff --cc src/java/org/apache/cassandra/auth/jmx/AuthorizationProxy.java index 7bfbf52,000..ebc1763 mode 100644,00..100644 --- a/src/java/org/apache/cassandra/auth/jmx/AuthorizationProxy.java +++ b/src/java/org/apache/cassandra/auth/jmx/AuthorizationProxy.java @@@ -1,494 -1,0 +1,493 @@@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.cassandra.auth.jmx; + +import java.lang.reflect.*; +import java.security.AccessControlContext; +import java.security.AccessController; +import java.security.Principal; +import java.util.Set; +import java.util.function.Function; +import java.util.function.Supplier; +import java.util.stream.Collectors; +import javax.management.MBeanServer; +import javax.management.MalformedObjectNameException; +import javax.management.ObjectName; +import javax.security.auth.Subject; + +import com.google.common.annotations.VisibleForTesting; - import com.google.common.base.Throwables; +import com.google.common.collect.ImmutableSet; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import org.apache.cassandra.auth.*; +import org.apache.cassandra.config.DatabaseDescriptor; +import org.apache.cassandra.service.StorageService; + +/** + * Provides a proxy interface to the platform's MBeanServer instance to perform + * role-based authorization on method invocation. + * + * When used in conjunction with a suitable JMXAuthenticator, which attaches a CassandraPrincipal + * to authenticated Subjects, this class uses the configured IAuthorizer to verify that the + * subject has the required permissions to execute methods on the MBeanServer and the MBeans it + * manages. + * + * Because an ObjectName may contain wildcards, meaning it represents a set of individual MBeans, + * JMX resources don't fit well with the hierarchical approach modelled by other IResource + * implementations and utilised by ClientState::ensureHasPermission etc. To enable grants to use + * pattern-type ObjectNames, this class performs its own custom matching and filtering of resources + * rather than pushing that down to the configured IAuthorizer. To that end, during authorization + * it pulls back all permissions for the active subject, filtering them to retain only grants on + * JMXResources. It then uses ObjectName::apply to assert whether the target MBeans are wholly + * represented by the resources with permissions. This means that
[cassandra] branch cassandra-3.0 updated (c092c46 -> 0a1e8d1)
This is an automated email from the ASF dual-hosted git repository.
samt pushed a change to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git.
from c092c46 Merge branch 'cassandra-2.2' into cassandra-3.0
new c8c3c26 Fix nomenclature of deny and allow lists
new 0a1e8d1 Merge branch 'cassandra-2.2' into cassandra-3.0
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CHANGES.txt| 1 +
.../cassandra/cql3/functions/UDFunction.java | 22
src/java/org/apache/cassandra/db/Directories.java | 28 ++---
...Directories.java => DisallowedDirectories.java} | 29 +++---
...sMBean.java => DisallowedDirectoriesMBean.java} | 3 ++-
.../db/compaction/AbstractCompactionStrategy.java | 6 ++---
.../cassandra/db/compaction/LeveledManifest.java | 2 +-
.../cassandra/hints/HintsDispatchExecutor.java | 2 +-
.../org/apache/cassandra/hints/HintsStore.java | 10
.../cassandra/io/sstable/format/SSTableReader.java | 2 +-
.../cassandra/service/DefaultFSErrorHandler.java | 6 ++---
test/unit/org/apache/cassandra/Util.java | 4 +--
.../org/apache/cassandra/db/DirectoriesTest.java | 2 +-
java => CorruptedSSTablesCompactionsTest.java} | 17 +++--
14 files changed, 69 insertions(+), 65 deletions(-)
rename src/java/org/apache/cassandra/db/{BlacklistedDirectories.java =>
DisallowedDirectories.java} (77%)
rename src/java/org/apache/cassandra/db/{BlacklistedDirectoriesMBean.java =>
DisallowedDirectoriesMBean.java} (95%)
rename
test/unit/org/apache/cassandra/db/compaction/{BlacklistingCompactionsTest.java
=> CorruptedSSTablesCompactionsTest.java} (93%)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated (259c63f -> be572f2)
This is an automated email from the ASF dual-hosted git repository.
samt pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git.
from 259c63f update version of Jackson to 2.9.10
new c8c3c26 Fix nomenclature of deny and allow lists
new 0a1e8d1 Merge branch 'cassandra-2.2' into cassandra-3.0
new 408f969 Merge branch 'cassandra-3.0' into cassandra-3.11
new be572f2 Merge branch 'cassandra-3.11' into trunk
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CHANGES.txt| 1 +
NEWS.txt | 8
.../cassandra/auth/jmx/AuthorizationProxy.java | 46 ++---
.../cassandra/cql3/functions/UDFunction.java | 22 +-
src/java/org/apache/cassandra/db/Directories.java | 47 +++---
...Directories.java => DisallowedDirectories.java} | 29 ++---
...sMBean.java => DisallowedDirectoriesMBean.java} | 2 +-
.../org/apache/cassandra/db/DiskBoundaries.java| 2 +-
.../apache/cassandra/db/DiskBoundaryManager.java | 6 +--
.../db/compaction/AbstractCompactionStrategy.java | 6 +--
.../cassandra/db/compaction/LeveledManifest.java | 2 +-
.../org/apache/cassandra/dht/RangeStreamer.java| 12 +++---
.../cassandra/hints/HintsDispatchExecutor.java | 2 +-
.../org/apache/cassandra/hints/HintsStore.java | 10 ++---
.../cassandra/io/sstable/format/SSTableReader.java | 2 +-
.../cassandra/service/DefaultFSErrorHandler.java | 6 +--
.../apache/cassandra/service/StorageService.java | 2 +-
test/unit/org/apache/cassandra/Util.java | 4 +-
.../cassandra/auth/jmx/AuthorizationProxyTest.java | 4 +-
.../org/apache/cassandra/db/DirectoriesTest.java | 2 +-
.../cassandra/db/DiskBoundaryManagerTest.java | 6 +--
java => CorruptedSSTablesCompactionsTest.java} | 17
22 files changed, 124 insertions(+), 114 deletions(-)
rename src/java/org/apache/cassandra/db/{BlacklistedDirectories.java =>
DisallowedDirectories.java} (80%)
rename src/java/org/apache/cassandra/db/{BlacklistedDirectoriesMBean.java =>
DisallowedDirectoriesMBean.java} (96%)
rename
test/unit/org/apache/cassandra/db/compaction/{BlacklistingCompactionsTest.java
=> CorruptedSSTablesCompactionsTest.java} (94%)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-3.11' into trunk
This is an automated email from the ASF dual-hosted git repository. samt pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit be572f27416e9477d5ac996087951ecdcd19e25a Merge: 259c63f 408f969 Author: Sam Tunnicliffe AuthorDate: Fri Jun 12 11:28:03 2020 +0100 Merge branch 'cassandra-3.11' into trunk CHANGES.txt| 1 + NEWS.txt | 8 .../cassandra/auth/jmx/AuthorizationProxy.java | 46 ++--- .../cassandra/cql3/functions/UDFunction.java | 22 +- src/java/org/apache/cassandra/db/Directories.java | 47 +++--- ...Directories.java => DisallowedDirectories.java} | 29 ++--- ...sMBean.java => DisallowedDirectoriesMBean.java} | 2 +- .../org/apache/cassandra/db/DiskBoundaries.java| 2 +- .../apache/cassandra/db/DiskBoundaryManager.java | 6 +-- .../db/compaction/AbstractCompactionStrategy.java | 6 +-- .../cassandra/db/compaction/LeveledManifest.java | 2 +- .../org/apache/cassandra/dht/RangeStreamer.java| 12 +++--- .../cassandra/hints/HintsDispatchExecutor.java | 2 +- .../org/apache/cassandra/hints/HintsStore.java | 10 ++--- .../cassandra/io/sstable/format/SSTableReader.java | 2 +- .../cassandra/service/DefaultFSErrorHandler.java | 6 +-- .../apache/cassandra/service/StorageService.java | 2 +- test/unit/org/apache/cassandra/Util.java | 4 +- .../cassandra/auth/jmx/AuthorizationProxyTest.java | 4 +- .../org/apache/cassandra/db/DirectoriesTest.java | 2 +- .../cassandra/db/DiskBoundaryManagerTest.java | 6 +-- java => CorruptedSSTablesCompactionsTest.java} | 17 22 files changed, 124 insertions(+), 114 deletions(-) diff --cc CHANGES.txt index 0c4203b,7f54146..5f09cdc --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -55,7 -15,10 +55,8 @@@ Merged from 3.0 * Allow selecting static column only when querying static index (CASSANDRA-14242) * cqlsh return non-zero status when STDIN CQL fails (CASSANDRA-15623) * Don't skip sstables in slice queries based only on local min/max/deletion timestamp (CASSANDRA-15690) - * Memtable memory allocations may deadlock (CASSANDRA-15367) - * Run evictFromMembership in GossipStage (CASSANDRA-15592) Merged from 2.2: + * Fix nomenclature of allow and deny lists (CASSANDRA-15862) * Remove generated files from source artifact (CASSANDRA-15849) * Remove duplicated tools binaries from tarballs (CASSANDRA-15768) * Duplicate results with DISTINCT queries in mixed mode (CASSANDRA-15501) diff --cc NEWS.txt index 8f18659,077bd8b..351f80d --- a/NEWS.txt +++ b/NEWS.txt @@@ -116,152 -47,8 +116,160 @@@ New feature Upgrading - -- Nothing specific to this release, but please see previous upgrading sections, - especially if you are upgrading from 3.0. +- Sstables for tables using with a frozen UDT written by C* 3.0 appear as corrupted. + + Background: The serialization-header in the -Statistics.db sstable component contains the type information + of the table columns. C* 3.0 write incorrect type information for frozen UDTs by omitting the + "frozen" information. Non-frozen UDTs were introduced by CASSANDRA-7423 in C* 3.6. Since then, the missing + "frozen" information leads to deserialization issues that result in CorruptSSTableExceptions, potentially other + exceptions as well. + + As a mitigation, the sstable serialization-headers are rewritten to contain the missing "frozen" information for + UDTs once, when an upgrade from C* 3.0 is detected. This migration does not touch snapshots or backups. + + The sstablescrub tool now performs a check of the sstable serialization-header against the schema. A mismatch of + the types in the serialization-header and the schema will cause sstablescrub to error out and stop by default. + See the new `-e` option. `-e off` disables the new validation code. `-e fix` or `-e fix-only`, e.g. + `sstablescrub -e fix keyspace table`, will validate the serialization-header, rewrite the non-frozen UDTs + in the serialzation-header to frozen UDTs, if that matches the schema, and continue with scrub. + See `sstablescrub -h`. + (CASSANDRA-15035) +- CASSANDRA-13241 lowered the default chunk_lengh_in_kb for compresesd tables from + 64kb to 16kb. For highly compressible data this can have a noticeable impact + on space utilization. You may want to consider manually specifying this value. +- Additional columns have been added to system_distributed.repair_history, + system_traces.sessions and system_traces.events. As a result select queries + against these tables - including queries against tracing tables performed + automatically by the drivers and cqlsh - will fail and generate an error in the log + during upgrade
[cassandra] branch cassandra-3.11 updated (e498539 -> 408f969)
This is an automated email from the ASF dual-hosted git repository.
samt pushed a change to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git.
from e498539 Merge branch 'cassandra-3.0' into cassandra-3.11
new c8c3c26 Fix nomenclature of deny and allow lists
new 0a1e8d1 Merge branch 'cassandra-2.2' into cassandra-3.0
new 408f969 Merge branch 'cassandra-3.0' into cassandra-3.11
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CHANGES.txt| 1 +
.../cassandra/auth/jmx/AuthorizationProxy.java | 47 +++---
.../cassandra/cql3/functions/UDFunction.java | 22 +-
src/java/org/apache/cassandra/db/Directories.java | 41 +--
...Directories.java => DisallowedDirectories.java} | 29 ++---
...sMBean.java => DisallowedDirectoriesMBean.java} | 2 +-
.../org/apache/cassandra/db/DiskBoundaries.java| 2 +-
.../apache/cassandra/db/DiskBoundaryManager.java | 6 +--
.../db/compaction/AbstractCompactionStrategy.java | 6 +--
.../cassandra/db/compaction/LeveledManifest.java | 2 +-
.../org/apache/cassandra/dht/RangeStreamer.java| 10 ++---
.../cassandra/hints/HintsDispatchExecutor.java | 2 +-
.../org/apache/cassandra/hints/HintsStore.java | 10 ++---
.../cassandra/io/sstable/format/SSTableReader.java | 2 +-
.../cassandra/service/DefaultFSErrorHandler.java | 6 +--
.../apache/cassandra/service/StorageService.java | 2 +-
test/unit/org/apache/cassandra/Util.java | 4 +-
.../cassandra/auth/jmx/AuthorizationProxyTest.java | 4 +-
.../org/apache/cassandra/db/DirectoriesTest.java | 2 +-
.../cassandra/db/DiskBoundaryManagerTest.java | 6 +--
java => CorruptedSSTablesCompactionsTest.java} | 17
21 files changed, 112 insertions(+), 111 deletions(-)
rename src/java/org/apache/cassandra/db/{BlacklistedDirectories.java =>
DisallowedDirectories.java} (80%)
rename src/java/org/apache/cassandra/db/{BlacklistedDirectoriesMBean.java =>
DisallowedDirectoriesMBean.java} (96%)
rename
test/unit/org/apache/cassandra/db/compaction/{BlacklistingCompactionsTest.java
=> CorruptedSSTablesCompactionsTest.java} (94%)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-2.2' into cassandra-3.0
This is an automated email from the ASF dual-hosted git repository.
samt pushed a commit to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 0a1e8d168bac0f562774814a808e313e1d2d6571
Merge: c092c46 c8c3c26
Author: Sam Tunnicliffe
AuthorDate: Fri Jun 12 11:21:14 2020 +0100
Merge branch 'cassandra-2.2' into cassandra-3.0
CHANGES.txt| 1 +
.../cassandra/cql3/functions/UDFunction.java | 22
src/java/org/apache/cassandra/db/Directories.java | 28 ++---
...Directories.java => DisallowedDirectories.java} | 29 +++---
...sMBean.java => DisallowedDirectoriesMBean.java} | 3 ++-
.../db/compaction/AbstractCompactionStrategy.java | 6 ++---
.../cassandra/db/compaction/LeveledManifest.java | 2 +-
.../cassandra/hints/HintsDispatchExecutor.java | 2 +-
.../org/apache/cassandra/hints/HintsStore.java | 10
.../cassandra/io/sstable/format/SSTableReader.java | 2 +-
.../cassandra/service/DefaultFSErrorHandler.java | 6 ++---
test/unit/org/apache/cassandra/Util.java | 4 +--
.../org/apache/cassandra/db/DirectoriesTest.java | 2 +-
java => CorruptedSSTablesCompactionsTest.java} | 17 +++--
14 files changed, 69 insertions(+), 65 deletions(-)
diff --cc CHANGES.txt
index 3fdbb96,b10a057..d506dc8
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,19 -1,5 +1,20 @@@
-2.2.18
+3.0.21
+ * Fix replica-side filtering returning stale data with CL > ONE
(CASSANDRA-8272, CASSANDRA-8273)
+ * Fix duplicated row on 2.x upgrades when multi-rows range tombstones
interact with collection ones (CASSANDRA-15805)
+ * Rely on snapshotted session infos on StreamResultFuture.maybeComplete to
avoid race conditions (CASSANDRA-15667)
+ * EmptyType doesn't override writeValue so could attempt to write bytes when
expected not to (CASSANDRA-15790)
+ * Fix index queries on partition key columns when some partitions contains
only static data (CASSANDRA-13666)
+ * Avoid creating duplicate rows during major upgrades (CASSANDRA-15789)
+ * liveDiskSpaceUsed and totalDiskSpaceUsed get corrupted if
IndexSummaryRedistribution gets interrupted (CASSANDRA-15674)
+ * Fix Debian init start/stop (CASSANDRA-15770)
+ * Fix infinite loop on index query paging in tables with clustering
(CASSANDRA-14242)
+ * Fix chunk index overflow due to large sstable with small chunk length
(CASSANDRA-15595)
+ * cqlsh return non-zero status when STDIN CQL fails (CASSANDRA-15623)
+ * Don't skip sstables in slice queries based only on local min/max/deletion
timestamp (CASSANDRA-15690)
+ * Memtable memory allocations may deadlock (CASSANDRA-15367)
+ * Run evictFromMembership in GossipStage (CASSANDRA-15592)
+Merged from 2.2:
+ * Fix nomenclature of allow and deny lists (CASSANDRA-15862)
* Remove generated files from source artifact (CASSANDRA-15849)
* Remove duplicated tools binaries from tarballs (CASSANDRA-15768)
* Duplicate results with DISTINCT queries in mixed mode (CASSANDRA-15501)
diff --cc src/java/org/apache/cassandra/cql3/functions/UDFunction.java
index 7b69342,1e5cea6..27f9eb8
--- a/src/java/org/apache/cassandra/cql3/functions/UDFunction.java
+++ b/src/java/org/apache/cassandra/cql3/functions/UDFunction.java
@@@ -71,111 -49,10 +71,111 @@@ public abstract class UDFunction extend
protected final String language;
protected final String body;
-protected final DataType[] argDataTypes;
-protected final DataType returnDataType;
+protected final TypeCodec[] argCodecs;
+protected final TypeCodec returnCodec;
protected final boolean calledOnNullInput;
+//
- // Access to classes is controlled via a whitelist and a blacklist.
++// Access to classes is controlled via allow and disallow lists.
+//
+// When a class is requested (both during compilation and runtime),
- // the whitelistedPatterns array is searched first, whether the
++// the allowedPatterns array is searched first, whether the
+// requested name matches one of the patterns. If not, nothing is
+// returned from the class-loader - meaning ClassNotFoundException
+// during runtime and "type could not resolved" during compilation.
+//
- // If a whitelisted pattern has been found, the blacklistedPatterns
++// If an allowed pattern has been found, the disallowedPatterns
+// array is searched for a match. If a match is found, class-loader
+// rejects access. Otherwise the class/resource can be loaded.
+//
- private static final String[] whitelistedPatterns =
++private static final String[] allowedPatterns =
+{
+"com/datastax/driver/core/",
+"com/google/common/reflect/TypeToken",
+"java/io/IOException.class",
+"java/io/Serializable.class",
+"java/lang/",
+"java/math/",
+"java/net/InetAddress.class",
+"java/net/Inet4Address.class",
+
[cassandra] branch cassandra-2.2 updated: Fix nomenclature of deny and allow lists
This is an automated email from the ASF dual-hosted git repository.
samt pushed a commit to branch cassandra-2.2
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cassandra-2.2 by this push:
new c8c3c26 Fix nomenclature of deny and allow lists
c8c3c26 is described below
commit c8c3c269bf1df944b73e2d4bf0008a6f412bf121
Author: Sam Tunnicliffe
AuthorDate: Thu Jun 11 18:40:18 2020 +0100
Fix nomenclature of deny and allow lists
Patch by Sam Tunnicliffe; reviewed by Jordan West for CASSANDRA-15862
---
CHANGES.txt| 1 +
src/java/org/apache/cassandra/db/Directories.java | 24 +-
...Directories.java => DisallowedDirectories.java} | 29 +++---
...sMBean.java => DisallowedDirectoriesMBean.java} | 3 ++-
.../db/compaction/AbstractCompactionStrategy.java | 6 ++---
.../cassandra/db/compaction/LeveledManifest.java | 2 +-
.../cassandra/io/sstable/format/SSTableReader.java | 2 +-
.../cassandra/service/DefaultFSErrorHandler.java | 6 ++---
.../org/apache/cassandra/cql3/OutOfSpaceBase.java | 4 +--
.../org/apache/cassandra/db/DirectoriesTest.java | 2 +-
java => CorruptedSSTablesCompactionsTest.java} | 15 ++-
11 files changed, 48 insertions(+), 46 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 788b2bf..b10a057 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.2.18
+ * Fix nomenclature of allow and deny lists (CASSANDRA-15862)
* Remove generated files from source artifact (CASSANDRA-15849)
* Remove duplicated tools binaries from tarballs (CASSANDRA-15768)
* Duplicate results with DISTINCT queries in mixed mode (CASSANDRA-15501)
diff --git a/src/java/org/apache/cassandra/db/Directories.java
b/src/java/org/apache/cassandra/db/Directories.java
index fa76b61..d1aa650 100644
--- a/src/java/org/apache/cassandra/db/Directories.java
+++ b/src/java/org/apache/cassandra/db/Directories.java
@@ -292,10 +292,10 @@ public class Directories
/**
* Basically the same as calling {@link #getWriteableLocationAsFile(long)}
with an unknown size ({@code -1L}),
- * which may return any non-blacklisted directory - even a data directory
that has no usable space.
+ * which may return any allowed directory - even a data directory that has
no usable space.
* Do not use this method in production code.
*
- * @throws IOError if all directories are blacklisted.
+ * @throws IOError if all directories are blocked.
*/
public File getDirectoryForNewSSTables()
{
@@ -303,9 +303,9 @@ public class Directories
}
/**
- * Returns a non-blacklisted data directory that _currently_ has {@code
writeSize} bytes as usable space.
+ * Returns an allowed directory that _currently_ has {@code writeSize}
bytes as usable space.
*
- * @throws IOError if all directories are blacklisted.
+ * @throws IOError if all directories are disallowed.
*/
public File getWriteableLocationAsFile(long writeSize)
{
@@ -313,9 +313,9 @@ public class Directories
}
/**
- * Returns a non-blacklisted data directory that _currently_ has {@code
writeSize} bytes as usable space.
+ * Returns an allowed data directory that _currently_ has {@code
writeSize} bytes as usable space.
*
- * @throws IOError if all directories are blacklisted.
+ * @throws IOError if all directories are disallowed.
*/
public DataDirectory getWriteableLocation(long writeSize)
{
@@ -323,13 +323,13 @@ public class Directories
long totalAvailable = 0L;
-// pick directories with enough space and so that resulting sstable
dirs aren't blacklisted for writes.
+// pick directories with enough space and so that resulting sstable
dirs aren't disallowed for writes.
boolean tooBig = false;
for (DataDirectory dataDir : dataDirectories)
{
-if
(BlacklistedDirectories.isUnwritable(getLocationForDisk(dataDir)))
+if
(DisallowedDirectories.isUnwritable(getLocationForDisk(dataDir)))
{
-logger.trace("removing blacklisted candidate {}",
dataDir.location);
+logger.trace("removing disallowed candidate {}",
dataDir.location);
continue;
}
DataDirectoryCandidate candidate = new
DataDirectoryCandidate(dataDir);
@@ -348,7 +348,7 @@ public class Directories
if (tooBig)
throw new RuntimeException("Insufficient disk space to write "
+ writeSize + " bytes");
else
-throw new FSWriteError(new IOException("All configured data
directories have been blacklisted as unwritable for erroring out"), "");
+throw new FSWriteError(new IOException("All configured data
directories have been disallowed as unwritable for erroring out"), "");
[jira] [Created] (CASSANDRA-15871) Cassandra driver first connection get the user's own schema information
maxwellguo created CASSANDRA-15871: -- Summary: Cassandra driver first connection get the user's own schema information Key: CASSANDRA-15871 URL: https://issues.apache.org/jira/browse/CASSANDRA-15871 Project: Cassandra Issue Type: Improvement Components: Cluster/Schema, Messaging/Client Reporter: maxwellguo We know that cassandra driver making a conenction with the coordinator node first time , the driver may select all the keyspaces/tables/columns/types from the server and cache the data locally. For different users they may have different tables and types ，so It is not suitable to get all the meta data cached , It is fine to just cache the user's own schema information not all. And doing this is safe and save first time connection resourse. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15869) MemoryOutputStream overflow on large bloom filter
[
https://issues.apache.org/jira/browse/CASSANDRA-15869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134016#comment-17134016
]
ZhaoYang commented on CASSANDRA-15869:
--
[~brandon.williams] do you mind reviewing?
> MemoryOutputStream overflow on large bloom filter
> -
>
> Key: CASSANDRA-15869
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15869
> Project: Cassandra
> Issue Type: Bug
> Components: Local/SSTable
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 4.0-beta
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Since CASSANDRA-9067, Cassandra will use {{MemoryOutputStream}} to
> reconstruct BF Memory without re-ordering bytes, see
> {{OffsetBitSet#deserialize}}. But {{MemoryOutputStream}} use {{INT}} to track
> position and will overflow when BF size exceeds 2GB.
> {code:java|title=stacktrace}
> error: Illegal bounds [-2147483648..-2147483584); size: 4588588016
> -- StackTrace --
> java.lang.AssertionError: Illegal bounds [-2147483648..-2147483584); size:
> 4588588016
> at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:185)
> at org.apache.cassandra.io.util.Memory.setBytes(Memory.java:138)
> at
> org.apache.cassandra.io.util.MemoryOutputStream.write(MemoryOutputStream.java:45)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15752) Range read concurrency factor didn't consider range merger
[
https://issues.apache.org/jira/browse/CASSANDRA-15752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhaoYang updated CASSANDRA-15752:
-
Test and Documentation Plan:
Added unit tests.
Circle Ci:
[https://circleci.com/workflow-run/440126c4-058d-4511-8301-16df2bddf5db]
was:
Added unit tests.
Circle Ci:
[https://circleci.com/workflow-run/cfcdbee3-c982-4ff7-ab87-e1b4c36cfdcd]
> Range read concurrency factor didn't consider range merger
> --
>
> Key: CASSANDRA-15752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15752
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Coordination
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
>
> During range read, coordinator computes concurrency factor which is the
> number of vnode ranges to contact in parallel for the next batch.
> But in {{RangeCommandIterator}}, vnode ranges are merged by {{RangeMerger}}
> if vnode ranges share enough replicas to satisfy consistency level. eg. vnode
> range [a,b) has replica n1,n2,n3 and vnode range [b,c) has replica n2,n3,n4,
> so they can be merged as range [a,c) with replica n2, n3 for Quorum.
> Currently it counts number of merged ranges towards concurrency factor.
> Coordinator may fetch more ranges than needed.
>
> Another issue is that when executing range read on table with very small
> amount of data, concurrency factor can be bumped to {{size of total vnode
> ranges}}, eg. 10k, depending on the num of vnodes and cluster size. As a
> result, coordinator will send large number of concurrent range requests,
> potentially slowing down the cluster.. We should cap the max concurrency
> factor..
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15867) Update Jackson version to 2.9.10.1 because there are security issues in 2.9.5
[ https://issues.apache.org/jira/browse/CASSANDRA-15867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17133974#comment-17133974 ] Stefan Miklosovic commented on CASSANDRA-15867: --- [~brandon.williams] dont we want to see this merged as well in 3.11? I see that coordinates for jackson are totally different so maybe I ll do that scan against 3.11 and see how we stand. > Update Jackson version to 2.9.10.1 because there are security issues in 2.9.5 > - > > Key: CASSANDRA-15867 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15867 > Project: Cassandra > Issue Type: Task > Components: Dependencies >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0-alpha5 > > Attachments: dependency-check-report.html > > > Please see attached HTML report from OWASP dependency check for current > 4.0-alpha5 trunk branch. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
