(cassandra) branch trunk updated (1163dd9a08 -> 9cc688e978)
This is an automated email from the ASF dual-hosted git repository. smiklosovic pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git from 1163dd9a08 Refactor cqlshmain global constants add 98d0e181d5 Remove bashisms for mx4j tool in cassandra-env.sh add 6250ba7929 Merge branch 'cassandra-4.0' into cassandra-4.1 add 5ae1294cb1 Merge branch 'cassandra-4.1' into cassandra-5.0 new 9cc688e978 Merge branch 'cassandra-5.0' into trunk The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt | 1 + conf/cassandra-env.sh | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) 01/01: Merge branch 'cassandra-5.0' into trunk
This is an automated email from the ASF dual-hosted git repository. smiklosovic pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 9cc688e978826baa8e44a78b3a46cea3a2aa1c5a Merge: 1163dd9a08 5ae1294cb1 Author: Stefan Miklosovic AuthorDate: Wed Feb 21 09:15:27 2024 +0100 Merge branch 'cassandra-5.0' into trunk CHANGES.txt | 1 + conf/cassandra-env.sh | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --cc CHANGES.txt index d2f959ee48,bb93d9d802..1e31dcdc0d --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -37,10 -17,14 +37,11 @@@ Merged from 5.0 * Fix resource cleanup after SAI query timeouts (CASSANDRA-19177) * Suppress CVE-2023-6481 (CASSANDRA-19184) Merged from 4.1: - * Memoize Cassandra verion and add a backoff interval for failed schema pulls (CASSANDRA-18902) * Fix StackOverflowError on ALTER after many previous schema changes (CASSANDRA-19166) + * Memoize Cassandra verion (CASSANDRA-18902) Merged from 4.0: + * Remove bashisms for mx4j tool in cassandra-env.sh (CASSANDRA-19416) * Add new concurrent_merkle_tree_requests config property to prevent OOM during multi-range and/or multi-table repairs (CASSANDRA-19336) - * Skip version check if an endpoint is dead state in Gossiper#upgradeFromVersionSupplier (CASSANDRA-19187) - * Fix Gossiper::hasMajorVersion3Nodes to return false during minor upgrade (CASSANDRA-18999) - * Revert unnecessary read lock acquisition when reading ring version in TokenMetadata introduced in CASSANDRA-16286 (CASSANDRA-19107) Merged from 3.11: Merged from 3.0: * Backport CASSANDRA-16418 to 3.x (CASSANDRA-18824) diff --cc conf/cassandra-env.sh index 0b1dec45ed,ba9f9d4596..4e28e687f9 --- a/conf/cassandra-env.sh +++ b/conf/cassandra-env.sh @@@ -271,8 -271,13 +271,8 @@@ JVM_OPTS="$JVM_OPTS -Dcom.sun.managemen #MX4J_ADDRESS="127.0.0.1" #MX4J_PORT="8081" -# Cassandra uses SIGAR to capture OS metrics CASSANDRA-7838 -# for SIGAR we have to set the java.library.path -# to the location of the native libraries. -JVM_OPTS="$JVM_OPTS -Djava.library.path=$CASSANDRA_HOME/lib/sigar-bin" - if [ "x$MX4J_ADDRESS" != "x" ]; then - if [[ "$MX4J_ADDRESS" == \-Dmx4jaddress* ]]; then + if [ "$(echo "$MX4J_ADDRESS" | grep -c "\-Dmx4jaddress")" = "1" ]; then # Backward compatible with the older style #13578 JVM_OPTS="$JVM_OPTS $MX4J_ADDRESS" else - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch cassandra-4.1 updated (a9a7dd0caf -> 6250ba7929)
This is an automated email from the ASF dual-hosted git repository. smiklosovic pushed a change to branch cassandra-4.1 in repository https://gitbox.apache.org/repos/asf/cassandra.git from a9a7dd0caf increment version to 4.1.5 add 98d0e181d5 Remove bashisms for mx4j tool in cassandra-env.sh add 6250ba7929 Merge branch 'cassandra-4.0' into cassandra-4.1 No new revisions were added by this update. Summary of changes: CHANGES.txt | 1 + conf/cassandra-env.sh | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch cassandra-4.0 updated (505f5af645 -> 98d0e181d5)
This is an automated email from the ASF dual-hosted git repository. smiklosovic pushed a change to branch cassandra-4.0 in repository https://gitbox.apache.org/repos/asf/cassandra.git from 505f5af645 Ensure that repair doesn't exceed repair_session_space by running limiting validation parallelism add 98d0e181d5 Remove bashisms for mx4j tool in cassandra-env.sh No new revisions were added by this update. Summary of changes: CHANGES.txt | 1 + conf/cassandra-env.sh | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch cassandra-5.0 updated (2aa2b59d37 -> 5ae1294cb1)
This is an automated email from the ASF dual-hosted git repository. smiklosovic pushed a change to branch cassandra-5.0 in repository https://gitbox.apache.org/repos/asf/cassandra.git from 2aa2b59d37 Minor adds and fixes in the documentation add 98d0e181d5 Remove bashisms for mx4j tool in cassandra-env.sh add 6250ba7929 Merge branch 'cassandra-4.0' into cassandra-4.1 add 5ae1294cb1 Merge branch 'cassandra-4.1' into cassandra-5.0 No new revisions were added by this update. Summary of changes: CHANGES.txt | 1 + conf/cassandra-env.sh | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19416) fix "if" condition for mx4j tool in cassandra-env.sh
[
https://issues.apache.org/jira/browse/CASSANDRA-19416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819139#comment-17819139
]
Stefan Miklosovic commented on CASSANDRA-19416:
---
I manually verified this works with /bin/bash as shebang too.
> fix "if" condition for mx4j tool in cassandra-env.sh
>
>
> Key: CASSANDRA-19416
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19416
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Tools
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There is this in cassandra-env.sh
> {code}
> if [[ "$MX4J_ADDRESS" == \-Dmx4jaddress* ]]; then
> {code}
> (similar for port)
> This is wrong for /bin/sh shell (our shebang in bin/cassandra) and this does
> not work, probably in bash only, because /bin/sh does not understand what
> "[[" is nor it understand what "==" is.
> The reason this was never detected so far is that the logic will never come
> there when MX4J_ADDRESS and / or MX4J_PORT is commented out couple lines
> above.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19414) Skinny dev circle workflow
[ https://issues.apache.org/jira/browse/CASSANDRA-19414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Berenguer Blasi updated CASSANDRA-19414: Change Category: Quality Assurance Complexity: Normal Component/s: CI Status: Open (was: Triage Needed) > Skinny dev circle workflow > -- > > Key: CASSANDRA-19414 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19414 > Project: Cassandra > Issue Type: Improvement > Components: CI >Reporter: Berenguer Blasi >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 5.0.x, 5.x > > > CircleCi CI runs are getting pretty heavy. During dev iterations we trigger > many CI pre-commit jobs which are just an overkill. > This ticket has the purpose to purge from the pre-commit workflow all > variations of the test matrix but the vanilla one. That should enable us for > a quick and cheap to iterate *during dev*, this is not a substitute for > pre-commit . This ticket's work will serve as the basis for the upcoming > changes being discussed > [atm|https://lists.apache.org/thread/qf5c3hhz6qkpyqvbd3sppzlmftlc0bw0] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19414) Skinny dev circle workflow
[ https://issues.apache.org/jira/browse/CASSANDRA-19414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Berenguer Blasi updated CASSANDRA-19414: Test and Documentation Plan: See PR Status: Patch Available (was: Open) > Skinny dev circle workflow > -- > > Key: CASSANDRA-19414 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19414 > Project: Cassandra > Issue Type: Improvement > Components: CI >Reporter: Berenguer Blasi >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 5.0.x, 5.x > > > CircleCi CI runs are getting pretty heavy. During dev iterations we trigger > many CI pre-commit jobs which are just an overkill. > This ticket has the purpose to purge from the pre-commit workflow all > variations of the test matrix but the vanilla one. That should enable us for > a quick and cheap to iterate *during dev*, this is not a substitute for > pre-commit . This ticket's work will serve as the basis for the upcoming > changes being discussed > [atm|https://lists.apache.org/thread/qf5c3hhz6qkpyqvbd3sppzlmftlc0bw0] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19416) fix "if" condition for mx4j tool in cassandra-env.sh
[
https://issues.apache.org/jira/browse/CASSANDRA-19416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-19416:
-
Status: Ready to Commit (was: Review In Progress)
> fix "if" condition for mx4j tool in cassandra-env.sh
>
>
> Key: CASSANDRA-19416
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19416
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Tools
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There is this in cassandra-env.sh
> {code}
> if [[ "$MX4J_ADDRESS" == \-Dmx4jaddress* ]]; then
> {code}
> (similar for port)
> This is wrong for /bin/sh shell (our shebang in bin/cassandra) and this does
> not work, probably in bash only, because /bin/sh does not understand what
> "[[" is nor it understand what "==" is.
> The reason this was never detected so far is that the logic will never come
> there when MX4J_ADDRESS and / or MX4J_PORT is commented out couple lines
> above.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19416) fix "if" condition for mx4j tool in cassandra-env.sh
[
https://issues.apache.org/jira/browse/CASSANDRA-19416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic updated CASSANDRA-19416:
--
Fix Version/s: 4.0.12
4.1.5
5.0-beta2
5.1
(was: 5.x)
(was: 4.0.x)
(was: 4.1.x)
(was: 5.0.x)
Since Version: 4.0
Source Control Link:
https://github.com/apache/cassandra/commit/98d0e181d53827b8459f8b0b4f013ed73328b2f1
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
> fix "if" condition for mx4j tool in cassandra-env.sh
>
>
> Key: CASSANDRA-19416
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19416
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Tools
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0.12, 4.1.5, 5.0-beta2, 5.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There is this in cassandra-env.sh
> {code}
> if [[ "$MX4J_ADDRESS" == \-Dmx4jaddress* ]]; then
> {code}
> (similar for port)
> This is wrong for /bin/sh shell (our shebang in bin/cassandra) and this does
> not work, probably in bash only, because /bin/sh does not understand what
> "[[" is nor it understand what "==" is.
> The reason this was never detected so far is that the logic will never come
> there when MX4J_ADDRESS and / or MX4J_PORT is commented out couple lines
> above.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19416) fix "if" condition for mx4j tool in cassandra-env.sh
[
https://issues.apache.org/jira/browse/CASSANDRA-19416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic updated CASSANDRA-19416:
--
Resolution: Fixed
Status: Resolved (was: Open)
> fix "if" condition for mx4j tool in cassandra-env.sh
>
>
> Key: CASSANDRA-19416
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19416
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Tools
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0.12, 4.1.5, 5.0-beta2, 5.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There is this in cassandra-env.sh
> {code}
> if [[ "$MX4J_ADDRESS" == \-Dmx4jaddress* ]]; then
> {code}
> (similar for port)
> This is wrong for /bin/sh shell (our shebang in bin/cassandra) and this does
> not work, probably in bash only, because /bin/sh does not understand what
> "[[" is nor it understand what "==" is.
> The reason this was never detected so far is that the logic will never come
> there when MX4J_ADDRESS and / or MX4J_PORT is commented out couple lines
> above.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19416) fix "if" condition for mx4j tool in cassandra-env.sh
[
https://issues.apache.org/jira/browse/CASSANDRA-19416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic updated CASSANDRA-19416:
--
Reviewers: Brandon Williams
> fix "if" condition for mx4j tool in cassandra-env.sh
>
>
> Key: CASSANDRA-19416
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19416
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Tools
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0.12, 4.1.5, 5.0-beta2, 5.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There is this in cassandra-env.sh
> {code}
> if [[ "$MX4J_ADDRESS" == \-Dmx4jaddress* ]]; then
> {code}
> (similar for port)
> This is wrong for /bin/sh shell (our shebang in bin/cassandra) and this does
> not work, probably in bash only, because /bin/sh does not understand what
> "[[" is nor it understand what "==" is.
> The reason this was never detected so far is that the logic will never come
> there when MX4J_ADDRESS and / or MX4J_PORT is commented out couple lines
> above.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19416) fix "if" condition for mx4j tool in cassandra-env.sh
[
https://issues.apache.org/jira/browse/CASSANDRA-19416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic updated CASSANDRA-19416:
--
Resolution: (was: Fixed)
Status: Open (was: Resolved)
> fix "if" condition for mx4j tool in cassandra-env.sh
>
>
> Key: CASSANDRA-19416
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19416
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Tools
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0.12, 4.1.5, 5.0-beta2, 5.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There is this in cassandra-env.sh
> {code}
> if [[ "$MX4J_ADDRESS" == \-Dmx4jaddress* ]]; then
> {code}
> (similar for port)
> This is wrong for /bin/sh shell (our shebang in bin/cassandra) and this does
> not work, probably in bash only, because /bin/sh does not understand what
> "[[" is nor it understand what "==" is.
> The reason this was never detected so far is that the logic will never come
> there when MX4J_ADDRESS and / or MX4J_PORT is commented out couple lines
> above.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19412) delete useless collection:backPressureHosts in the sendToHintedReplicas to improve write performance
[
https://issues.apache.org/jira/browse/CASSANDRA-19412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819204#comment-17819204
]
Stefan Miklosovic commented on CASSANDRA-19412:
---
[CASSANDRA-19412-4.1|https://github.com/instaclustr/cassandra/tree/CASSANDRA-19412-4.1]
{noformat}
java11_pre-commit_tests
java11_separate_tests
java8_pre-commit_tests
✓ j8_build 7m 42s
✓ j8_cqlsh_dtests_py3 8m 52s
✓ j8_cqlsh_dtests_py3116m 22s
✓ j8_cqlsh_dtests_py311_vnode 5m 59s
✓ j8_cqlsh_dtests_py38 7m 3s
✓ j8_cqlsh_dtests_py38_vnode 9m 10s
✓ j8_cqlsh_dtests_py3_vnode8m 27s
✓ j8_cqlshlib_cython_tests 9m 23s
✓ j8_cqlshlib_tests 11m 25s
✓ j8_dtests 31m 17s
✓ j8_dtests_vnode 35m 40s
✓ j8_jvm_dtests 15m 47s
✓ j8_jvm_dtests_vnode 12m 31s
✓ j8_simulator_dtests 1m 52s
✓ j11_jvm_dtests_vnode12m 34s
✓ j11_jvm_dtests 15m 33s
✓ j11_dtests_vnode36m 49s
✓ j11_dtests 33m 19s
✓ j11_cqlshlib_tests 5m 57s
✓ j11_cqlshlib_cython_tests6m 36s
✓ j11_cqlsh_dtests_py3_vnode 5m 24s
✓ j11_cqlsh_dtests_py38_vnode 5m 42s
✓ j11_cqlsh_dtests_py385m 40s
✓ j11_cqlsh_dtests_py311_vnode 5m 34s
✓ j11_cqlsh_dtests_py311 5m 53s
✓ j11_cqlsh_dtests_py3 5m 18s
✕ j8_unit_tests 10m 53s
org.apache.cassandra.cql3.MemtableSizeTest testSize[skiplist]
✕ j8_utests_system_keyspace_directory 10m 37s
org.apache.cassandra.cql3.MemtableSizeTest testSize[skiplist]
✕ j11_unit_tests 9m 49s
org.apache.cassandra.net.ConnectionTest testTimeout
org.apache.cassandra.cql3.MemtableSizeTest testSize[skiplist]
java8_separate_tests
{noformat}
[java11_pre-commit_tests|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3899/workflows/8f32f197-4fd4-48c8-a68a-bd91de42d4ba]
[java11_separate_tests|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3899/workflows/c18e72df-6393-40c4-b5e0-a02d48b363c9]
[java8_pre-commit_tests|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3899/workflows/915a2c34-3e6d-44a2-baef-9b32c79b2026]
[java8_separate_tests|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3899/workflows/4fcf7df2-fda2-4644-92e2-af0bb2181e55]
> delete useless collection:backPressureHosts in the sendToHintedReplicas to
> improve write performance
>
>
> Key: CASSANDRA-19412
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19412
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Local Write-Read Paths
>Reporter: Ling Mao
>Assignee: Ling Mao
>Priority: Low
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Every normal write request will go through this
> method({_}*sendToHintedReplicas*{_}). However, the list:backPressureHosts in
> the method has never been used functionally.
> The _*backpressure*_ was introduced by:
> {code:java}
> Support optional backpressure strategies at the coordinator
> patch by Sergio Bossa; reviewed by Stefania Alborghetti for CASSANDRA-9318
> d43b9ce5 Sergio Bossa on 2016/9/19 at 10:42 AM {code}
> {code:java}
> public static void sendToHintedEndpoints(final Mutation mutation,
> Iterable targets,
>
> AbstractWriteResponseHandler responseHandler,
> String localDataCenter,
> Stage stage)
> throws OverloadedException
> {
> int targetsSize = Iterables.size(targets);
> // this dc replicas:
> Collection localDc = null;
> // extra-datacenter replicas, grouped by dc
> Map> dcGroups = null;
> // only need to create a Message for non-local writes
> MessageOut message = null;
> boolean insertLocal = false;
> ArrayList endpointsToHint = null;
> List b
[jira] [Created] (CASSANDRA-19417) LIST SUPERUSERS cql command
Shailaja Koppu created CASSANDRA-19417: -- Summary: LIST SUPERUSERS cql command Key: CASSANDRA-19417 URL: https://issues.apache.org/jira/browse/CASSANDRA-19417 Project: Cassandra Issue Type: Improvement Components: Tool/cqlsh Reporter: Shailaja Koppu Assignee: Shailaja Koppu Developing a new CQL command LIST SUPERUSERS to return list of roles with superuser privilege. This includes roles who acquired superuser privilege in the hierarchy. Context: LIST ROLES cql command lists roles, their membership details and displays super=true for immediate superusers. But there can be roles who acquired superuser privilege due to a grant. LIST ROLES command won't display super=true for such roles and the only way to recognize such roles is to look for atleast one row with super=true in the output of LIST ROLES OF command. While this works to check is a given role has superuser privilege, there may be services (for example, Sidecar) working with C* and may need to maintain list of roles with superuser privilege. There is no existing command/tool to retrieve such roles details. Hence developing this command which returns all roles having superuser privilege. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19412) delete useless collection:backPressureHosts in the sendToHintedReplicas to improve write performance
[
https://issues.apache.org/jira/browse/CASSANDRA-19412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819305#comment-17819305
]
Stefan Miklosovic commented on CASSANDRA-19412:
---
[CASSANDRA-19412-4.0|https://github.com/instaclustr/cassandra/tree/CASSANDRA-19412-4.0]
{noformat}
java11_pre-commit_tests
java11_separate_tests
java8_pre-commit_tests
✓ j8_build 1m 42s
✓ j8_cqlsh-dtests-py2-no-vnodes8m 24s
✓ j8_cqlsh-dtests-py2-with-vnodes 8m 30s
✓ j8_cqlsh_dtests_py3 8m 10s
✓ j8_cqlsh_dtests_py3119m 10s
✓ j8_cqlsh_dtests_py311_vnode 8m 40s
✓ j8_cqlsh_dtests_py38 6m 15s
✓ j8_cqlsh_dtests_py38_vnode 5m 31s
✓ j8_cqlsh_dtests_py3_vnode 7m 2s
✓ j8_cqlshlib_tests 10m 11s
✓ j8_dtests 31m 30s
✓ j8_dtests_vnode 35m 25s
✓ j8_jvm_dtests 13m 20s
✓ j11_dtests_vnode33m 46s
✓ j11_dtests 32m 30s
✓ j11_cqlsh_dtests_py3_vnode 5m 22s
✓ j11_cqlsh_dtests_py38_vnode 5m 57s
✓ j11_cqlsh_dtests_py385m 16s
✓ j11_cqlsh_dtests_py311_vnode 5m 38s
✓ j11_cqlsh_dtests_py311 5m 34s
✓ j11_cqlsh_dtests_py3 5m 23s
✓ j11_cqlsh-dtests-py2-with-vnodes 5m 27s
✓ j11_cqlsh-dtests-py2-no-vnodes 5m 35s
✕ j8_unit_tests8m 49s
org.apache.cassandra.cql3.MemtableSizeTest testTruncationReleasesLogSpace
✕ j8_utests_system_keyspace_directory 9m 11s
org.apache.cassandra.cql3.MemtableSizeTest testTruncationReleasesLogSpace
✕ j11_unit_tests 7m 29s
org.apache.cassandra.cql3.MemtableSizeTest testTruncationReleasesLogSpace
java8_separate_tests
{noformat}
[java11_pre-commit_tests|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3908/workflows/a6669c63-6c96-4ca1-913c-16bf757696fd]
[java11_separate_tests|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3908/workflows/391a8c83-1918-42bd-80f6-b0e40f5dd161]
[java8_pre-commit_tests|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3908/workflows/8c0c8ba8-2e46-458e-a68b-1de9be3e733a]
[java8_separate_tests|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3908/workflows/31ce1e70-6430-4383-b7f0-5d5d83989477]
> delete useless collection:backPressureHosts in the sendToHintedReplicas to
> improve write performance
>
>
> Key: CASSANDRA-19412
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19412
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Local Write-Read Paths
>Reporter: Ling Mao
>Assignee: Ling Mao
>Priority: Low
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Every normal write request will go through this
> method({_}*sendToHintedReplicas*{_}). However, the list:backPressureHosts in
> the method has never been used functionally.
> The _*backpressure*_ was introduced by:
> {code:java}
> Support optional backpressure strategies at the coordinator
> patch by Sergio Bossa; reviewed by Stefania Alborghetti for CASSANDRA-9318
> d43b9ce5 Sergio Bossa on 2016/9/19 at 10:42 AM {code}
> {code:java}
> public static void sendToHintedEndpoints(final Mutation mutation,
> Iterable targets,
>
> AbstractWriteResponseHandler responseHandler,
> String localDataCenter,
> Stage stage)
> throws OverloadedException
> {
> int targetsSize = Iterables.size(targets);
> // this dc replicas:
> Collection localDc = null;
> // extra-datacenter replicas, grouped by dc
> Map> dcGroups = null;
> // only need to create a Message for non-local writes
> MessageOut message = null;
> boolean insertLocal = false;
> ArrayList endpointsToHint = null;
> List backPressureHosts = null;
> for (InetAddress destination : targets)
> {
> checkHintOverload(destination);
> if (FailureDetector.instance.isAlive(destination))
>
[jira] [Updated] (CASSANDRA-19344) Test Failure: org.apache.cassandra.distributed.test.TransientRangeMovementTest.testRemoveNode-_jdk17
[
https://issues.apache.org/jira/browse/CASSANDRA-19344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe updated CASSANDRA-19344:
Epic Link: CASSANDRA-19055
> Test Failure:
> org.apache.cassandra.distributed.test.TransientRangeMovementTest.testRemoveNode-_jdk17
>
>
> Key: CASSANDRA-19344
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19344
> Project: Cassandra
> Issue Type: Bug
> Components: CI
>Reporter: Ekaterina Dimitrova
>Assignee: Sam Tunnicliffe
>Priority: Normal
> Fix For: 5.x
>
>
> The test can fail in two different ways:
> {code:java}
> junit.framework.AssertionFailedError: NOT IN CURRENT: 31 -- [(00,20),
> (31,50)] at
> org.apache.cassandra.distributed.test.TransientRangeMovementTest.assertAllContained(TransientRangeMovementTest.java:203)
> at
> org.apache.cassandra.distributed.test.TransientRangeMovementTest.testRemoveNode(TransientRangeMovementTest.java:183)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
> as in here -
> [https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2639/workflows/32b92ce7-5e9d-4efb-8362-d200d2414597/jobs/55139/tests#failed-test-0]
> and
> {code:java}
> junit.framework.AssertionFailedError: nodetool command [removenode,
> 6d194555-f6eb-41d0-c000-0003, --force] was not successful stdout:
> stderr: error: Node /127.0.0.4:7012 is alive and owns this ID. Use
> decommission command to remove it from the ring -- StackTrace --
> java.lang.UnsupportedOperationException: Node /127.0.0.4:7012 is alive and
> owns this ID. Use decommission command to remove it from the ring at
> org.apache.cassandra.tcm.sequences.SingleNodeSequences.removeNode(SingleNodeSequences.java:110)
> at
> org.apache.cassandra.service.StorageService.removeNode(StorageService.java:3682)
> at org.apache.cassandra.tools.NodeProbe.removeNode(NodeProbe.java:1020) at
> org.apache.cassandra.tools.nodetool.RemoveNode.execute(RemoveNode.java:51) at
> org.apache.cassandra.tools.NodeTool$NodeToolCmd.runInternal(NodeTool.java:388)
> at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:373) at
> org.apache.cassandra.tools.NodeTool.execute(NodeTool.java:272) at
> org.apache.cassandra.distributed.impl.Instance$DTestNodeTool.execute(Instance.java:1129)
> at
> org.apache.cassandra.distributed.impl.Instance.lambda$nodetoolResult$51(Instance.java:1038)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61) at
> org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71) at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:833) Notifications: Error:
> java.lang.UnsupportedOperationException: Node /127.0.0.4:7012 is alive and
> owns this ID. Use decommission command to remove it from the ring at
> org.apache.cassandra.tcm.sequences.SingleNodeSequences.removeNode(SingleNodeSequences.java:110)
> at
> org.apache.cassandra.service.StorageService.removeNode(StorageService.java:3682)
> at org.apache.cassandra.tools.NodeProbe.removeNode(NodeProbe.java:1020) at
> org.apache.cassandra.tools.nodetool.RemoveNode.execute(RemoveNode.java:51) at
> org.apache.cassandra.tools.NodeTool$NodeToolCmd.runInternal(NodeTool.java:388)
> at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:373) at
> org.apache.cassandra.tools.NodeTool.execute(NodeTool.java:272) at
> org.apache.cassandra.distributed.impl.Instance$DTestNodeTool.execute(Instance.java:1129)
> at
> org.apache.cassandra.distributed.impl.Instance.lambda$nodetoolResult$51(Instance.java:1038)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61) at
> org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71) at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:833) at
> org.apache.cassandra.distributed.api.NodeToolResult$Asserts.fail(NodeToolResult.java:214)
> at
> org.apache.cassandra.distributed.api.NodeToolResult$Asser
[jira] [Commented] (CASSANDRA-19018) An SAI-specific mechanism to ensure consistency isn't violated for multi-column (i.e. AND) queries at CL > ONE
[
https://issues.apache.org/jira/browse/CASSANDRA-19018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819326#comment-17819326
]
Andres de la Peña commented on CASSANDRA-19018:
---
The changes look good to me, +1. Here is 5.0 CI with repeated runs for the new
and modified tests:
||PR||CI||
|[5.0|https://github.com/apache/cassandra/pull/2935]|[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/3443/workflows/11c238f4-5b28-4a37-a653-1016d9d1dfa1]
[j17|https://app.circleci.com/pipelines/github/adelapena/cassandra/3443/workflows/84d857b5-6097-40dd-a276-772b443bf9de]|
> An SAI-specific mechanism to ensure consistency isn't violated for
> multi-column (i.e. AND) queries at CL > ONE
> --
>
> Key: CASSANDRA-19018
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19018
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Coordination, Feature/SAI
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
> Fix For: 5.0-rc, 5.x
>
> Attachments: ci_summary-1.html, ci_summary.html,
> result_details.tar-1.gz, result_details.tar.gz
>
> Time Spent: 10h 50m
> Remaining Estimate: 0h
>
> CASSANDRA-19007 is going to be where we add a guardrail around
> filtering/index queries that use intersection/AND over partially updated
> non-key columns. (ex. Restricting one clustering column and one normal column
> does not cause a consistency problem, as primary keys cannot be partially
> updated.) This issue exists to attempt to fix this specifically for SAI in
> 5.0.x, as Accord will (last I checked) not be available until the 5.1 release.
> The SAI-specific version of the originally reported issue is this:
> {noformat}
> try (Cluster cluster = init(Cluster.build(2).withConfig(config ->
> config.with(GOSSIP).with(NETWORK)).start()))
> {
> cluster.schemaChange(withKeyspace("CREATE TABLE %s.t (k int
> PRIMARY KEY, a int, b int)"));
> cluster.schemaChange(withKeyspace("CREATE INDEX ON %s.t(a) USING
> 'sai'"));
> cluster.schemaChange(withKeyspace("CREATE INDEX ON %s.t(b) USING
> 'sai'"));
> // insert a split row
> cluster.get(1).executeInternal(withKeyspace("INSERT INTO %s.t(k,
> a) VALUES (0, 1)"));
> cluster.get(2).executeInternal(withKeyspace("INSERT INTO %s.t(k,
> b) VALUES (0, 2)"));
> // Uncomment this line and test succeeds w/ partial writes
> completed...
> //cluster.get(1).nodetoolResult("repair",
> KEYSPACE).asserts().success();
> String select = withKeyspace("SELECT * FROM %s.t WHERE a = 1 AND
> b = 2");
> Object[][] initialRows = cluster.coordinator(1).execute(select,
> ConsistencyLevel.ALL);
> assertRows(initialRows, row(0, 1, 2)); // not found!!
> }
> {noformat}
> To make a long story short, the local SAI indexes are hiding local partial
> matches from the coordinator that would combine there to form full matches.
> Simple non-index filtering queries also suffer from this problem, but they
> hide the partial matches in a different way. I'll outline a possible solution
> for this in the comments that takes advantage of replica filtering protection
> and the repaired/unrepaired datasets...and attempts to minimize the amount of
> extra row data sent to the coordinator.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19018) An SAI-specific mechanism to ensure consistency isn't violated for multi-column (i.e. AND) queries at CL > ONE
[
https://issues.apache.org/jira/browse/CASSANDRA-19018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819327#comment-17819327
]
Caleb Rackliffe commented on CASSANDRA-19018:
-
Nice! Looks like Circle more or less confirms what I'm seeing on my CI...clean
w/ the exception of CASSANDRA-19168
> An SAI-specific mechanism to ensure consistency isn't violated for
> multi-column (i.e. AND) queries at CL > ONE
> --
>
> Key: CASSANDRA-19018
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19018
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Coordination, Feature/SAI
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
> Fix For: 5.0-rc, 5.x
>
> Attachments: ci_summary-1.html, ci_summary.html,
> result_details.tar-1.gz, result_details.tar.gz
>
> Time Spent: 10h 50m
> Remaining Estimate: 0h
>
> CASSANDRA-19007 is going to be where we add a guardrail around
> filtering/index queries that use intersection/AND over partially updated
> non-key columns. (ex. Restricting one clustering column and one normal column
> does not cause a consistency problem, as primary keys cannot be partially
> updated.) This issue exists to attempt to fix this specifically for SAI in
> 5.0.x, as Accord will (last I checked) not be available until the 5.1 release.
> The SAI-specific version of the originally reported issue is this:
> {noformat}
> try (Cluster cluster = init(Cluster.build(2).withConfig(config ->
> config.with(GOSSIP).with(NETWORK)).start()))
> {
> cluster.schemaChange(withKeyspace("CREATE TABLE %s.t (k int
> PRIMARY KEY, a int, b int)"));
> cluster.schemaChange(withKeyspace("CREATE INDEX ON %s.t(a) USING
> 'sai'"));
> cluster.schemaChange(withKeyspace("CREATE INDEX ON %s.t(b) USING
> 'sai'"));
> // insert a split row
> cluster.get(1).executeInternal(withKeyspace("INSERT INTO %s.t(k,
> a) VALUES (0, 1)"));
> cluster.get(2).executeInternal(withKeyspace("INSERT INTO %s.t(k,
> b) VALUES (0, 2)"));
> // Uncomment this line and test succeeds w/ partial writes
> completed...
> //cluster.get(1).nodetoolResult("repair",
> KEYSPACE).asserts().success();
> String select = withKeyspace("SELECT * FROM %s.t WHERE a = 1 AND
> b = 2");
> Object[][] initialRows = cluster.coordinator(1).execute(select,
> ConsistencyLevel.ALL);
> assertRows(initialRows, row(0, 1, 2)); // not found!!
> }
> {noformat}
> To make a long story short, the local SAI indexes are hiding local partial
> matches from the coordinator that would combine there to form full matches.
> Simple non-index filtering queries also suffer from this problem, but they
> hide the partial matches in a different way. I'll outline a possible solution
> for this in the comments that takes advantage of replica filtering protection
> and the repaired/unrepaired datasets...and attempts to minimize the amount of
> extra row data sent to the coordinator.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19406) Optimize pre-commit test suite
[ https://issues.apache.org/jira/browse/CASSANDRA-19406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819350#comment-17819350 ] Josh McKenzie commented on CASSANDRA-19406: --- I don't think we really came to a consensus on the dev ML threads did we? Or do we have a lazy consensus on this and are good to move forward? > Optimize pre-commit test suite > -- > > Key: CASSANDRA-19406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19406 > Project: Cassandra > Issue Type: Epic >Reporter: Jacek Lewandowski >Priority: Normal > > See the following ML threads: > * Branimir discussing having 2 supported configs which included a lot of > back and forth about what we're testing: > [link|https://lists.apache.org/thread/qf5c3hhz6qkpyqvbd3sppzlmftlc0bw0] > * Follow up DISCUSS thread notifying about this ticket: > [link|https://lists.apache.org/thread/r433j1mk111v102z4nr1m9l1fqf41fbr] > The pre-commit test suite has a lot of redundancy in our current approach > while also leaving certain configurations untested or undertested. We need to > address both sides of this. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19406) Optimize pre-commit test suite
[ https://issues.apache.org/jira/browse/CASSANDRA-19406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie updated CASSANDRA-19406: -- Description: See the following ML threads: * Branimir discussing having 2 supported configs which included a lot of back and forth about what we're testing: [link|https://lists.apache.org/thread/qf5c3hhz6qkpyqvbd3sppzlmftlc0bw0] * Follow up DISCUSS thread notifying about this ticket: [link|https://lists.apache.org/thread/r433j1mk111v102z4nr1m9l1fqf41fbr] The pre-commit test suite has a lot of redundancy in our current approach while also leaving certain configurations untested or undertested. We need to address both sides of this. > Optimize pre-commit test suite > -- > > Key: CASSANDRA-19406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19406 > Project: Cassandra > Issue Type: Epic >Reporter: Jacek Lewandowski >Priority: Normal > > See the following ML threads: > * Branimir discussing having 2 supported configs which included a lot of > back and forth about what we're testing: > [link|https://lists.apache.org/thread/qf5c3hhz6qkpyqvbd3sppzlmftlc0bw0] > * Follow up DISCUSS thread notifying about this ticket: > [link|https://lists.apache.org/thread/r433j1mk111v102z4nr1m9l1fqf41fbr] > The pre-commit test suite has a lot of redundancy in our current approach > while also leaving certain configurations untested or undertested. We need to > address both sides of this. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19412) delete useless collection:backPressureHosts in the sendToHintedReplicas to improve write performance
[
https://issues.apache.org/jira/browse/CASSANDRA-19412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819353#comment-17819353
]
Stefan Miklosovic commented on CASSANDRA-19412:
---
[CASSANDRA-19412-5.0|https://github.com/instaclustr/cassandra/tree/CASSANDRA-19412-5.0]
{noformat}
java17_pre-commit_tests
✓ j17_build4m 12s
✓ j17_cqlsh_dtests_py311 5m 43s
✓ j17_cqlsh_dtests_py311_vnode 6m 10s
✓ j17_cqlsh_dtests_py385m 54s
✓ j17_cqlsh_dtests_py38_vnode 6m 25s
✓ j17_cqlshlib_cython_tests7m 34s
✓ j17_cqlshlib_tests 6m 41s
✓ j17_dtests 34m 10s
✓ j17_dtests_vnode33m 26s
✓ j17_jvm_dtests 19m 26s
✕ j17_jvm_dtests_vnode17m 16s
org.apache.cassandra.distributed.test.ClearSnapshotTest
clearSnapshotSlowTest
✕ j17_unit_tests 14m 41s
org.apache.cassandra.index.sai.cql.VectorUpdateDeleteTest updateTest
✕ j17_utests_oa16m 8s
org.apache.cassandra.index.sai.cql.VectorUpdateDeleteTest updateTest
java17_separate_tests
java11_pre-commit_tests
java11_separate_tests
{noformat}
[java17_pre-commit_tests|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3909/workflows/651a6039-ea4c-4c4f-b25b-21f7b85ac889]
[java17_separate_tests|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3909/workflows/8fff875c-0962-4550-bac6-bdd8567dc932]
[java11_pre-commit_tests|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3909/workflows/dc229a55-f7aa-44ac-9634-ab3ef6007712]
[java11_separate_tests|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3909/workflows/afd7e775-7bab-40e2-aaf9-82321243f04f]
> delete useless collection:backPressureHosts in the sendToHintedReplicas to
> improve write performance
>
>
> Key: CASSANDRA-19412
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19412
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Local Write-Read Paths
>Reporter: Ling Mao
>Assignee: Ling Mao
>Priority: Low
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Every normal write request will go through this
> method({_}*sendToHintedReplicas*{_}). However, the list:backPressureHosts in
> the method has never been used functionally.
> The _*backpressure*_ was introduced by:
> {code:java}
> Support optional backpressure strategies at the coordinator
> patch by Sergio Bossa; reviewed by Stefania Alborghetti for CASSANDRA-9318
> d43b9ce5 Sergio Bossa on 2016/9/19 at 10:42 AM {code}
> {code:java}
> public static void sendToHintedEndpoints(final Mutation mutation,
> Iterable targets,
>
> AbstractWriteResponseHandler responseHandler,
> String localDataCenter,
> Stage stage)
> throws OverloadedException
> {
> int targetsSize = Iterables.size(targets);
> // this dc replicas:
> Collection localDc = null;
> // extra-datacenter replicas, grouped by dc
> Map> dcGroups = null;
> // only need to create a Message for non-local writes
> MessageOut message = null;
> boolean insertLocal = false;
> ArrayList endpointsToHint = null;
> List backPressureHosts = null;
> for (InetAddress destination : targets)
> {
> checkHintOverload(destination);
> if (FailureDetector.instance.isAlive(destination))
> {
> if (canDoLocalRequest(destination))
> {
> insertLocal = true;
> }
> else
> {
> // belongs on a different server
> if (message == null)
> message = mutation.createMessage();
> String dc =
> DatabaseDescriptor.getEndpointSnitch().getDatacenter(destination);
> // direct writes to local DC or old Cassandra versions
> // (1.1 knows how to forward old-style String message
> IDs; updated to int in 2.0)
> if (localDataCenter.equals(dc))
> {
> if (localDc == null)
>
[jira] [Updated] (CASSANDRA-19412) delete useless collection:backPressureHosts in the sendToHintedReplicas to improve write performance
[
https://issues.apache.org/jira/browse/CASSANDRA-19412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-19412:
-
Reviewers: Brandon Williams
Status: Review In Progress (was: Needs Committer)
> delete useless collection:backPressureHosts in the sendToHintedReplicas to
> improve write performance
>
>
> Key: CASSANDRA-19412
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19412
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Local Write-Read Paths
>Reporter: Ling Mao
>Assignee: Ling Mao
>Priority: Low
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Every normal write request will go through this
> method({_}*sendToHintedReplicas*{_}). However, the list:backPressureHosts in
> the method has never been used functionally.
> The _*backpressure*_ was introduced by:
> {code:java}
> Support optional backpressure strategies at the coordinator
> patch by Sergio Bossa; reviewed by Stefania Alborghetti for CASSANDRA-9318
> d43b9ce5 Sergio Bossa on 2016/9/19 at 10:42 AM {code}
> {code:java}
> public static void sendToHintedEndpoints(final Mutation mutation,
> Iterable targets,
>
> AbstractWriteResponseHandler responseHandler,
> String localDataCenter,
> Stage stage)
> throws OverloadedException
> {
> int targetsSize = Iterables.size(targets);
> // this dc replicas:
> Collection localDc = null;
> // extra-datacenter replicas, grouped by dc
> Map> dcGroups = null;
> // only need to create a Message for non-local writes
> MessageOut message = null;
> boolean insertLocal = false;
> ArrayList endpointsToHint = null;
> List backPressureHosts = null;
> for (InetAddress destination : targets)
> {
> checkHintOverload(destination);
> if (FailureDetector.instance.isAlive(destination))
> {
> if (canDoLocalRequest(destination))
> {
> insertLocal = true;
> }
> else
> {
> // belongs on a different server
> if (message == null)
> message = mutation.createMessage();
> String dc =
> DatabaseDescriptor.getEndpointSnitch().getDatacenter(destination);
> // direct writes to local DC or old Cassandra versions
> // (1.1 knows how to forward old-style String message
> IDs; updated to int in 2.0)
> if (localDataCenter.equals(dc))
> {
> if (localDc == null)
> localDc = new ArrayList<>(targetsSize);
> localDc.add(destination);
> }
> else
> {
> Collection messages = (dcGroups != null)
> ? dcGroups.get(dc) : null;
> if (messages == null)
> {
> messages = new ArrayList<>(3); // most DCs will
> have <= 3 replicas
> if (dcGroups == null)
> dcGroups = new HashMap<>();
> dcGroups.put(dc, messages);
> }
> messages.add(destination);
> }
> if (backPressureHosts == null)
> backPressureHosts = new ArrayList<>(targetsSize);
> backPressureHosts.add(destination);
> }
> }
> else
> {
> if (shouldHint(destination))
> {
> if (endpointsToHint == null)
> endpointsToHint = new ArrayList<>(targetsSize);
> endpointsToHint.add(destination);
> }
> }
> }
> if (backPressureHosts != null)
> MessagingService.instance().applyBackPressure(backPressureHosts,
> responseHandler.currentTimeout());
> if (endpointsToHint != null)
> submitHint(mutation, endpointsToHint, responseHandler);
> if (insertLocal)
> performLocally(stage, Optional.of(mutation), mutation::apply,
> responseHandler);
> if (localDc != null)
> {
> for (InetAddress destination : localDc)
> MessagingService.instance().sendRR(message, destination,
> responseHan
[jira] [Commented] (CASSANDRA-19412) delete useless collection:backPressureHosts in the sendToHintedReplicas to improve write performance
[
https://issues.apache.org/jira/browse/CASSANDRA-19412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819355#comment-17819355
]
Brandon Williams commented on CASSANDRA-19412:
--
+1 from me.
> delete useless collection:backPressureHosts in the sendToHintedReplicas to
> improve write performance
>
>
> Key: CASSANDRA-19412
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19412
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Local Write-Read Paths
>Reporter: Ling Mao
>Assignee: Ling Mao
>Priority: Low
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Every normal write request will go through this
> method({_}*sendToHintedReplicas*{_}). However, the list:backPressureHosts in
> the method has never been used functionally.
> The _*backpressure*_ was introduced by:
> {code:java}
> Support optional backpressure strategies at the coordinator
> patch by Sergio Bossa; reviewed by Stefania Alborghetti for CASSANDRA-9318
> d43b9ce5 Sergio Bossa on 2016/9/19 at 10:42 AM {code}
> {code:java}
> public static void sendToHintedEndpoints(final Mutation mutation,
> Iterable targets,
>
> AbstractWriteResponseHandler responseHandler,
> String localDataCenter,
> Stage stage)
> throws OverloadedException
> {
> int targetsSize = Iterables.size(targets);
> // this dc replicas:
> Collection localDc = null;
> // extra-datacenter replicas, grouped by dc
> Map> dcGroups = null;
> // only need to create a Message for non-local writes
> MessageOut message = null;
> boolean insertLocal = false;
> ArrayList endpointsToHint = null;
> List backPressureHosts = null;
> for (InetAddress destination : targets)
> {
> checkHintOverload(destination);
> if (FailureDetector.instance.isAlive(destination))
> {
> if (canDoLocalRequest(destination))
> {
> insertLocal = true;
> }
> else
> {
> // belongs on a different server
> if (message == null)
> message = mutation.createMessage();
> String dc =
> DatabaseDescriptor.getEndpointSnitch().getDatacenter(destination);
> // direct writes to local DC or old Cassandra versions
> // (1.1 knows how to forward old-style String message
> IDs; updated to int in 2.0)
> if (localDataCenter.equals(dc))
> {
> if (localDc == null)
> localDc = new ArrayList<>(targetsSize);
> localDc.add(destination);
> }
> else
> {
> Collection messages = (dcGroups != null)
> ? dcGroups.get(dc) : null;
> if (messages == null)
> {
> messages = new ArrayList<>(3); // most DCs will
> have <= 3 replicas
> if (dcGroups == null)
> dcGroups = new HashMap<>();
> dcGroups.put(dc, messages);
> }
> messages.add(destination);
> }
> if (backPressureHosts == null)
> backPressureHosts = new ArrayList<>(targetsSize);
> backPressureHosts.add(destination);
> }
> }
> else
> {
> if (shouldHint(destination))
> {
> if (endpointsToHint == null)
> endpointsToHint = new ArrayList<>(targetsSize);
> endpointsToHint.add(destination);
> }
> }
> }
> if (backPressureHosts != null)
> MessagingService.instance().applyBackPressure(backPressureHosts,
> responseHandler.currentTimeout());
> if (endpointsToHint != null)
> submitHint(mutation, endpointsToHint, responseHandler);
> if (insertLocal)
> performLocally(stage, Optional.of(mutation), mutation::apply,
> responseHandler);
> if (localDc != null)
> {
> for (InetAddress destination : localDc)
> MessagingService.instance().sendRR(message, destination,
> responseHandler, true);
>
[jira] [Commented] (CASSANDRA-19412) delete useless collection:backPressureHosts in the sendToHintedReplicas to improve write performance
[
https://issues.apache.org/jira/browse/CASSANDRA-19412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819357#comment-17819357
]
Stefan Miklosovic commented on CASSANDRA-19412:
---
hi [~maxwellguo] do you want to review as well?
> delete useless collection:backPressureHosts in the sendToHintedReplicas to
> improve write performance
>
>
> Key: CASSANDRA-19412
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19412
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Local Write-Read Paths
>Reporter: Ling Mao
>Assignee: Ling Mao
>Priority: Low
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Every normal write request will go through this
> method({_}*sendToHintedReplicas*{_}). However, the list:backPressureHosts in
> the method has never been used functionally.
> The _*backpressure*_ was introduced by:
> {code:java}
> Support optional backpressure strategies at the coordinator
> patch by Sergio Bossa; reviewed by Stefania Alborghetti for CASSANDRA-9318
> d43b9ce5 Sergio Bossa on 2016/9/19 at 10:42 AM {code}
> {code:java}
> public static void sendToHintedEndpoints(final Mutation mutation,
> Iterable targets,
>
> AbstractWriteResponseHandler responseHandler,
> String localDataCenter,
> Stage stage)
> throws OverloadedException
> {
> int targetsSize = Iterables.size(targets);
> // this dc replicas:
> Collection localDc = null;
> // extra-datacenter replicas, grouped by dc
> Map> dcGroups = null;
> // only need to create a Message for non-local writes
> MessageOut message = null;
> boolean insertLocal = false;
> ArrayList endpointsToHint = null;
> List backPressureHosts = null;
> for (InetAddress destination : targets)
> {
> checkHintOverload(destination);
> if (FailureDetector.instance.isAlive(destination))
> {
> if (canDoLocalRequest(destination))
> {
> insertLocal = true;
> }
> else
> {
> // belongs on a different server
> if (message == null)
> message = mutation.createMessage();
> String dc =
> DatabaseDescriptor.getEndpointSnitch().getDatacenter(destination);
> // direct writes to local DC or old Cassandra versions
> // (1.1 knows how to forward old-style String message
> IDs; updated to int in 2.0)
> if (localDataCenter.equals(dc))
> {
> if (localDc == null)
> localDc = new ArrayList<>(targetsSize);
> localDc.add(destination);
> }
> else
> {
> Collection messages = (dcGroups != null)
> ? dcGroups.get(dc) : null;
> if (messages == null)
> {
> messages = new ArrayList<>(3); // most DCs will
> have <= 3 replicas
> if (dcGroups == null)
> dcGroups = new HashMap<>();
> dcGroups.put(dc, messages);
> }
> messages.add(destination);
> }
> if (backPressureHosts == null)
> backPressureHosts = new ArrayList<>(targetsSize);
> backPressureHosts.add(destination);
> }
> }
> else
> {
> if (shouldHint(destination))
> {
> if (endpointsToHint == null)
> endpointsToHint = new ArrayList<>(targetsSize);
> endpointsToHint.add(destination);
> }
> }
> }
> if (backPressureHosts != null)
> MessagingService.instance().applyBackPressure(backPressureHosts,
> responseHandler.currentTimeout());
> if (endpointsToHint != null)
> submitHint(mutation, endpointsToHint, responseHandler);
> if (insertLocal)
> performLocally(stage, Optional.of(mutation), mutation::apply,
> responseHandler);
> if (localDc != null)
> {
> for (InetAddress destination : localDc)
> MessagingService.instance().sendRR(message, destinat
[jira] [Created] (CASSANDRA-19418) [Analytics] Report additional bulk analytics job stats for instrumentation
Arjun Ashok created CASSANDRA-19418: --- Summary: [Analytics] Report additional bulk analytics job stats for instrumentation Key: CASSANDRA-19418 URL: https://issues.apache.org/jira/browse/CASSANDRA-19418 Project: Cassandra Issue Type: Task Reporter: Arjun Ashok Currently, the Cassandra bulk analytics library supports a "dialHome" API to publish some initial job metadata, which in its current form, is redirected to a log. The intention behind this is to allow custom implementations that can utilize these summarized stats for instrumentation or reporting of client behavior. This task is meant to enhance this API to allow for additional job metadata to be published both at the Spark executor level and at the task levels to gather stats such as "success/failure", "number of rows written/read", "failure reason" etc. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19418) [Analytics] Report additional bulk analytics job stats for instrumentation
[ https://issues.apache.org/jira/browse/CASSANDRA-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Ashok updated CASSANDRA-19418: Change Category: Operability > [Analytics] Report additional bulk analytics job stats for instrumentation > -- > > Key: CASSANDRA-19418 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19418 > Project: Cassandra > Issue Type: Task > Components: Analytics Library >Reporter: Arjun Ashok >Priority: Normal > > Currently, the Cassandra bulk analytics library supports a "dialHome" API to > publish some initial job metadata, which in its current form, is redirected > to a log. The intention behind this is to allow custom implementations that > can utilize these summarized stats for instrumentation or reporting of client > behavior. > This task is meant to enhance this API to allow for additional job metadata > to be published both at the Spark executor level and at the task levels to > gather stats such as "success/failure", "number of rows written/read", > "failure reason" etc. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19418) [Analytics] Report additional bulk analytics job stats for instrumentation
[ https://issues.apache.org/jira/browse/CASSANDRA-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Ashok updated CASSANDRA-19418: Component/s: Analytics Library > [Analytics] Report additional bulk analytics job stats for instrumentation > -- > > Key: CASSANDRA-19418 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19418 > Project: Cassandra > Issue Type: Task > Components: Analytics Library >Reporter: Arjun Ashok >Priority: Normal > > Currently, the Cassandra bulk analytics library supports a "dialHome" API to > publish some initial job metadata, which in its current form, is redirected > to a log. The intention behind this is to allow custom implementations that > can utilize these summarized stats for instrumentation or reporting of client > behavior. > This task is meant to enhance this API to allow for additional job metadata > to be published both at the Spark executor level and at the task levels to > gather stats such as "success/failure", "number of rows written/read", > "failure reason" etc. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-19418) [Analytics] Report additional bulk analytics job stats for instrumentation
[ https://issues.apache.org/jira/browse/CASSANDRA-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Ashok reassigned CASSANDRA-19418: --- Assignee: Arjun Ashok > [Analytics] Report additional bulk analytics job stats for instrumentation > -- > > Key: CASSANDRA-19418 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19418 > Project: Cassandra > Issue Type: Task > Components: Analytics Library >Reporter: Arjun Ashok >Assignee: Arjun Ashok >Priority: Normal > > Currently, the Cassandra bulk analytics library supports a "dialHome" API to > publish some initial job metadata, which in its current form, is redirected > to a log. The intention behind this is to allow custom implementations that > can utilize these summarized stats for instrumentation or reporting of client > behavior. > This task is meant to enhance this API to allow for additional job metadata > to be published both at the Spark executor level and at the task levels to > gather stats such as "success/failure", "number of rows written/read", > "failure reason" etc. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[PR] CASSANDRA-19418 - Changes to report additional bulk analytics job stats for instrumentation [cassandra-analytics]
arjunashok opened a new pull request, #41: URL: https://github.com/apache/cassandra-analytics/pull/41 ## Changes - Update to relay the results of the spark tasks to the executor to instrument the total no. of rows written. - Abstracts the current `dialHome` implementation within `CassandraBulkWriterContext` to be pluggable via a `Reportable` interface, that can allow recording of job parameters both at the executor level and at the task level. - The above abstraction also separates the recording of the metadata as the job progresses and separately publish them at the end of the job with success/failure status and failure reason where applicable. ## Testing - Validated successful build run w/ unit/integration tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-19419) Non-transactional schema updates can interfere with Accord transaction execuion
Ariel Weisberg created CASSANDRA-19419: -- Summary: Non-transactional schema updates can interfere with Accord transaction execuion Key: CASSANDRA-19419 URL: https://issues.apache.org/jira/browse/CASSANDRA-19419 Project: Cassandra Issue Type: Bug Reporter: Ariel Weisberg While Accord can handle topology changes correctly it can’t handle non-transaction schema updates because those execute outside of Accord. When Accord tries to execute a transaction against the schema in the epoch the transaction is supposed to execute in then it is possible for different nodes to see different schemas when reading or writing data as part of a transaction. Dropping a needed a column or table is the most likely issue as we don't support altering column types. Because commit is async it is possible for a table or to be dropped before the writes can be propagated after it was acknowledged instead of signaling an error. While the table was dropped it's possible the client needed the error to know that the request was processed improperly or that it needed to take some other action client side. Or add table where the original coordinator can't read the table, but the recovery coordinator can and might apply different results to different replicas. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19284) Harry overrides model
[ https://issues.apache.org/jira/browse/CASSANDRA-19284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819392#comment-17819392 ] Caleb Rackliffe commented on CASSANDRA-19284: - Completed my review, with review left in the PR > Harry overrides model > - > > Key: CASSANDRA-19284 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19284 > Project: Cassandra > Issue Type: New Feature > Components: Test/fuzz >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: High > Attachments: ci_summary.html, result_details.tar.gz > > > Harry model to allow providing specific values for the test. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19411) [Analytics] Bulk reader fails to produce a row when regular column values are null
[
https://issues.apache.org/jira/browse/CASSANDRA-19411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yifan Cai updated CASSANDRA-19411:
--
Reviewers: Yifan Cai, Yifan Cai
Status: Review In Progress (was: Patch Available)
> [Analytics] Bulk reader fails to produce a row when regular column values are
> null
> --
>
> Key: CASSANDRA-19411
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19411
> Project: Cassandra
> Issue Type: Bug
> Components: Analytics Library
>Reporter: Francisco Guerrero
>Assignee: Francisco Guerrero
>Priority: Normal
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Bulk Reader won't emit a row when the regular column values are all null. For
> example,
> a schema {{PK}} = {{a,b}} ; {{CK}} = {{c,d}} ; and columns = {{e,f}}
> || a|| b || c|| d || e || f ||
> | {{pk1}} | {{pk2}} | {{ck1}} | {{ck2}} | {{null}} | {{null}} |
> when queried from bulk reader, it won't produce a row.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19411) [Analytics] Bulk reader fails to produce a row when regular column values are null
[
https://issues.apache.org/jira/browse/CASSANDRA-19411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yifan Cai updated CASSANDRA-19411:
--
Reviewers: Yifan Cai (was: Yifan Cai, Yifan Cai)
> [Analytics] Bulk reader fails to produce a row when regular column values are
> null
> --
>
> Key: CASSANDRA-19411
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19411
> Project: Cassandra
> Issue Type: Bug
> Components: Analytics Library
>Reporter: Francisco Guerrero
>Assignee: Francisco Guerrero
>Priority: Normal
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Bulk Reader won't emit a row when the regular column values are all null. For
> example,
> a schema {{PK}} = {{a,b}} ; {{CK}} = {{c,d}} ; and columns = {{e,f}}
> || a|| b || c|| d || e || f ||
> | {{pk1}} | {{pk2}} | {{ck1}} | {{ck2}} | {{null}} | {{null}} |
> when queried from bulk reader, it won't produce a row.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-19420) WEBSITE - Link fix for 5.0 landing page
Diogenese Topper created CASSANDRA-19420: Summary: WEBSITE - Link fix for 5.0 landing page Key: CASSANDRA-19420 URL: https://issues.apache.org/jira/browse/CASSANDRA-19420 Project: Cassandra Issue Type: Task Components: Documentation/Website Reporter: Diogenese Topper link fix for mathematical functions link -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra-analytics) branch trunk updated: CASSANDRA-19411: Bulk reader fails to produce a row when regular column values are null
This is an automated email from the ASF dual-hosted git repository.
frankgh pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra-analytics.git
The following commit(s) were added to refs/heads/trunk by this push:
new 46c35d0 CASSANDRA-19411: Bulk reader fails to produce a row when
regular column values are null
46c35d0 is described below
commit 46c35d0ef2efb66512133a7913df9936b0a80dc8
Author: Francisco Guerrero
AuthorDate: Mon Feb 19 20:50:16 2024 -0800
CASSANDRA-19411: Bulk reader fails to produce a row when regular column
values are null
Bulk Reader won't emit a row when the regular column values are all `null`.
For example,
a schema `PK` = `a`, `b` ; `CK` = `c`, `d` ; and columns = `e`, `f`.
| a | b | c | d | e | f |
| --- | --- | --- | --- | | |
| pk1 | pk2 | ck1 | ck2 | null | null |
When queried from Analytics bulk reader, it won't produce a row.
This issue also occurs when the projected regular column values are all
`null`, where
other non-projected columns might have some values.
Patch by Francisco Guerrero; Reviewed by Yifan Cai for CASSANDRA-19411
---
CHANGES.txt| 1 +
.../spark/sparksql/AbstractSparkRowIterator.java | 6 +-
.../cassandra/spark/sparksql/FullRowBuilder.java | 12 +-
.../spark/sparksql/SparkCellIterator.java | 51 +---
.../cassandra/spark/sparksql/SparkRowIterator.java | 2 +-
.../cassandra/spark/sparksql/SparkRowIterator.java | 16 +--
.../org/apache/cassandra/spark/EndToEndTests.java | 113
.../java/org/apache/cassandra/spark/Tester.java| 143 +
.../cassandra/spark/reader/EmptyStreamScanner.java | 6 +
.../cassandra/spark/reader/StreamScanner.java | 5 +
.../spark/reader/common/IndexIterator.java | 6 +
.../cassandra/spark/utils/test/TestSchema.java | 28 +++-
.../spark/reader/AbstractStreamScanner.java| 6 +
13 files changed, 271 insertions(+), 124 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 9daa5f0..1472baf 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
1.0.0
+ * Bulk reader fails to produce a row when regular column values are null
(CASSANDRA-19411)
* Use XXHash32 for digest calculation of SSTables (CASSANDRA-19369)
* Startup Validation Failures when Checking Sidecar Connectivity
(CASSANDRA-19377)
* No longer need to synchronize on Schema.instance after Cassandra 4.0.12
(CASSANDRA-19351)
diff --git
a/cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/sparksql/AbstractSparkRowIterator.java
b/cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/sparksql/AbstractSparkRowIterator.java
index 7882fe9..cc2f810 100644
---
a/cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/sparksql/AbstractSparkRowIterator.java
+++
b/cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/sparksql/AbstractSparkRowIterator.java
@@ -49,8 +49,8 @@ abstract class AbstractSparkRowIterator
protected final List requestedFeatures;
protected final CqlTable cqlTable;
-protected final boolean noValueColumns;
protected final StructType columnFilter;
+protected final boolean hasProjectedValueColumns;
private Cell cell = null;
private InternalRow row = null;
@@ -67,7 +67,7 @@ abstract class AbstractSparkRowIterator
this.stats.openedSparkRowIterator();
this.openTimeNanos = System.nanoTime();
this.requestedFeatures = dataLayer.requestedFeatures();
-this.noValueColumns = it.noValueColumns();
+this.hasProjectedValueColumns = it.hasProjectedValueColumns();
this.builder = newBuilder();
}
@@ -132,7 +132,7 @@ abstract class AbstractSparkRowIterator
builder.onCell(cell);
-if (!noValueColumns)
+if (hasProjectedValueColumns)
{
// If schema has value column
builder.copyValue(cell);
diff --git
a/cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/sparksql/FullRowBuilder.java
b/cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/sparksql/FullRowBuilder.java
index 823fdbe..5a06343 100644
---
a/cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/sparksql/FullRowBuilder.java
+++
b/cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/sparksql/FullRowBuilder.java
@@ -35,18 +35,18 @@ class FullRowBuilder implements RowBuilder
static final Object[] EMPTY_RESULT = new Object[0];
final int numColumns;
final int numCells;
-final boolean noValueColumns;
+final boolean hasProjectedValueColumns;
int extraColumns;
Object[] result;
int count;
private final CqlTable cqlTable;
-FullRowBuilder(CqlTable cqlTable, boolean noValueColumns)
+FullRowBuilder(CqlTable cqlTable, boolean ha
[jira] [Updated] (CASSANDRA-19411) [Analytics] Bulk reader fails to produce a row when regular column values are null
[
https://issues.apache.org/jira/browse/CASSANDRA-19411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francisco Guerrero updated CASSANDRA-19411:
---
Status: Ready to Commit (was: Review In Progress)
> [Analytics] Bulk reader fails to produce a row when regular column values are
> null
> --
>
> Key: CASSANDRA-19411
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19411
> Project: Cassandra
> Issue Type: Bug
> Components: Analytics Library
>Reporter: Francisco Guerrero
>Assignee: Francisco Guerrero
>Priority: Normal
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Bulk Reader won't emit a row when the regular column values are all null. For
> example,
> a schema {{PK}} = {{a,b}} ; {{CK}} = {{c,d}} ; and columns = {{e,f}}
> || a|| b || c|| d || e || f ||
> | {{pk1}} | {{pk2}} | {{ck1}} | {{ck2}} | {{null}} | {{null}} |
> when queried from bulk reader, it won't produce a row.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
Re: [PR] CASSANDRA-19411: Bulk reader fails to produce a row when regular column values are null [cassandra-analytics]
frankgh closed pull request #40: CASSANDRA-19411: Bulk reader fails to produce a row when regular column values are null URL: https://github.com/apache/cassandra-analytics/pull/40 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
Re: [PR] CASSANDRA-19411: Bulk reader fails to produce a row when regular column values are null [cassandra-analytics]
frankgh commented on PR #40: URL: https://github.com/apache/cassandra-analytics/pull/40#issuecomment-1958435849 Closed via https://github.com/apache/cassandra-analytics/commit/46c35d0ef2efb66512133a7913df9936b0a80dc8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19411) [Analytics] Bulk reader fails to produce a row when regular column values are null
[
https://issues.apache.org/jira/browse/CASSANDRA-19411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francisco Guerrero updated CASSANDRA-19411:
---
Fix Version/s: NA
Since Version: NA
Source Control Link:
https://github.com/apache/cassandra-analytics/commit/46c35d0ef2efb66512133a7913df9936b0a80dc8
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
> [Analytics] Bulk reader fails to produce a row when regular column values are
> null
> --
>
> Key: CASSANDRA-19411
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19411
> Project: Cassandra
> Issue Type: Bug
> Components: Analytics Library
>Reporter: Francisco Guerrero
>Assignee: Francisco Guerrero
>Priority: Normal
> Fix For: NA
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Bulk Reader won't emit a row when the regular column values are all null. For
> example,
> a schema {{PK}} = {{a,b}} ; {{CK}} = {{c,d}} ; and columns = {{e,f}}
> || a|| b || c|| d || e || f ||
> | {{pk1}} | {{pk2}} | {{ck1}} | {{ck2}} | {{null}} | {{null}} |
> when queried from bulk reader, it won't produce a row.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19420) WEBSITE - Link fix for mathematical functions on 5.0 landing page
[ https://issues.apache.org/jira/browse/CASSANDRA-19420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Diogenese Topper updated CASSANDRA-19420: - Change Category: Semantic Complexity: Normal Impacts: Docs (was: None) Test and Documentation Plan: correct the link Description: link fix for mathematical functions on 5.0 landing page (was: link fix for mathematical functions link) Summary: WEBSITE - Link fix for mathematical functions on 5.0 landing page (was: WEBSITE - Link fix for 5.0 landing page) > WEBSITE - Link fix for mathematical functions on 5.0 landing page > - > > Key: CASSANDRA-19420 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19420 > Project: Cassandra > Issue Type: Task > Components: Documentation/Website >Reporter: Diogenese Topper >Priority: Normal > > link fix for mathematical functions on 5.0 landing page -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19420) WEBSITE - Link fix for mathematical functions on 5.0 landing page
[ https://issues.apache.org/jira/browse/CASSANDRA-19420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated CASSANDRA-19420: --- Labels: pull-request-available (was: ) > WEBSITE - Link fix for mathematical functions on 5.0 landing page > - > > Key: CASSANDRA-19420 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19420 > Project: Cassandra > Issue Type: Task > Components: Documentation/Website >Reporter: Diogenese Topper >Priority: Normal > Labels: pull-request-available > > link fix for mathematical functions on 5.0 landing page -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19420) WEBSITE - Link fix for mathematical functions on 5.0 landing page
[ https://issues.apache.org/jira/browse/CASSANDRA-19420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Diogenese Topper updated CASSANDRA-19420: - Status: Open (was: Triage Needed) https://github.com/apache/cassandra-website/pull/266 > WEBSITE - Link fix for mathematical functions on 5.0 landing page > - > > Key: CASSANDRA-19420 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19420 > Project: Cassandra > Issue Type: Task > Components: Documentation/Website >Reporter: Diogenese Topper >Priority: Normal > Labels: pull-request-available > > link fix for mathematical functions on 5.0 landing page -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19336) Repair causes out of memory
[
https://issues.apache.org/jira/browse/CASSANDRA-19336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819427#comment-17819427
]
Manish Khandelwal commented on CASSANDRA-19336:
---
Fix Version should be 4.0.13 and probably 4.1.5.
> Repair causes out of memory
> ---
>
> Key: CASSANDRA-19336
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19336
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Repair
>Reporter: Andres de la Peña
>Assignee: Andres de la Peña
>Priority: Normal
> Fix For: 4.0.12, 4.1.4, 5.0-beta2, 5.1
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> CASSANDRA-14096 introduced {{repair_session_space}} as a limit for the memory
> usage for Merkle tree calculations during repairs. This limit is applied to
> the set of Merkle trees built for a received validation request
> ({{{}VALIDATION_REQ{}}}), divided by the replication factor so as not to
> overwhelm the repair coordinator, who will have requested RF sets of Merkle
> trees. That way the repair coordinator should only use
> {{repair_session_space}} for the RF Merkle trees.
> However, a repair session without {{{}-pr-{}}}/{{{}-partitioner-range{}}}
> will send RF*RF validation requests, because the repair coordinator node has
> RF-1 replicas and is also the replica of RF-1 nodes. Since all the requests
> are sent at the same time, at some point the repair coordinator can have up
> to RF*{{{}repair_session_space{}}} worth of Merkle trees if none of the
> validation responses is fully processed before the last response arrives.
> Even worse, if the cluster uses virtual nodes, many nodes can be replicas of
> the repair coordinator, and some nodes can be replicas of multiple token
> ranges. It would mean that the repair coordinator can send more than RF or
> RF*RF simultaneous validation requests.
> For example, in an 11-node cluster with RF=3 and 256 tokens, we have seen a
> repair session involving 44 groups of ranges to be repaired. This produces
> 44*3=132 validation requests contacting all the nodes in the cluster. When
> the responses for all these requests start to arrive to the coordinator, each
> containing up to {{repair_session_space}}/3 of Merkle trees, they accumulate
> quicker than they are consumed, greatly exceeding {{repair_session_space}}
> and OOMing the node.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19412) delete useless collection:backPressureHosts in the sendToHintedReplicas to improve write performance
[
https://issues.apache.org/jira/browse/CASSANDRA-19412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819438#comment-17819438
]
Maxwell Guo commented on CASSANDRA-19412:
-
+1 too
> delete useless collection:backPressureHosts in the sendToHintedReplicas to
> improve write performance
>
>
> Key: CASSANDRA-19412
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19412
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Local Write-Read Paths
>Reporter: Ling Mao
>Assignee: Ling Mao
>Priority: Low
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Every normal write request will go through this
> method({_}*sendToHintedReplicas*{_}). However, the list:backPressureHosts in
> the method has never been used functionally.
> The _*backpressure*_ was introduced by:
> {code:java}
> Support optional backpressure strategies at the coordinator
> patch by Sergio Bossa; reviewed by Stefania Alborghetti for CASSANDRA-9318
> d43b9ce5 Sergio Bossa on 2016/9/19 at 10:42 AM {code}
> {code:java}
> public static void sendToHintedEndpoints(final Mutation mutation,
> Iterable targets,
>
> AbstractWriteResponseHandler responseHandler,
> String localDataCenter,
> Stage stage)
> throws OverloadedException
> {
> int targetsSize = Iterables.size(targets);
> // this dc replicas:
> Collection localDc = null;
> // extra-datacenter replicas, grouped by dc
> Map> dcGroups = null;
> // only need to create a Message for non-local writes
> MessageOut message = null;
> boolean insertLocal = false;
> ArrayList endpointsToHint = null;
> List backPressureHosts = null;
> for (InetAddress destination : targets)
> {
> checkHintOverload(destination);
> if (FailureDetector.instance.isAlive(destination))
> {
> if (canDoLocalRequest(destination))
> {
> insertLocal = true;
> }
> else
> {
> // belongs on a different server
> if (message == null)
> message = mutation.createMessage();
> String dc =
> DatabaseDescriptor.getEndpointSnitch().getDatacenter(destination);
> // direct writes to local DC or old Cassandra versions
> // (1.1 knows how to forward old-style String message
> IDs; updated to int in 2.0)
> if (localDataCenter.equals(dc))
> {
> if (localDc == null)
> localDc = new ArrayList<>(targetsSize);
> localDc.add(destination);
> }
> else
> {
> Collection messages = (dcGroups != null)
> ? dcGroups.get(dc) : null;
> if (messages == null)
> {
> messages = new ArrayList<>(3); // most DCs will
> have <= 3 replicas
> if (dcGroups == null)
> dcGroups = new HashMap<>();
> dcGroups.put(dc, messages);
> }
> messages.add(destination);
> }
> if (backPressureHosts == null)
> backPressureHosts = new ArrayList<>(targetsSize);
> backPressureHosts.add(destination);
> }
> }
> else
> {
> if (shouldHint(destination))
> {
> if (endpointsToHint == null)
> endpointsToHint = new ArrayList<>(targetsSize);
> endpointsToHint.add(destination);
> }
> }
> }
> if (backPressureHosts != null)
> MessagingService.instance().applyBackPressure(backPressureHosts,
> responseHandler.currentTimeout());
> if (endpointsToHint != null)
> submitHint(mutation, endpointsToHint, responseHandler);
> if (insertLocal)
> performLocally(stage, Optional.of(mutation), mutation::apply,
> responseHandler);
> if (localDc != null)
> {
> for (InetAddress destination : localDc)
> MessagingService.instance().sendRR(message, destination,
> responseHandler, true);
> }
>
[jira] [Commented] (CASSANDRA-19406) Optimize pre-commit test suite
[ https://issues.apache.org/jira/browse/CASSANDRA-19406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819487#comment-17819487 ] Berenguer Blasi commented on CASSANDRA-19406: - Nobody seems to be opposing the first part: having a skinny dev circle workflow which is totally opt-in CASSANDRA-19414. So I would think lazy consensus can be applied and I started a discuss thread for added visibility. Once CASSANDRA-18753 merges we can revisit the pre-commit workflows as any agreements will have been reached by then. Does it make sense? > Optimize pre-commit test suite > -- > > Key: CASSANDRA-19406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19406 > Project: Cassandra > Issue Type: Epic >Reporter: Jacek Lewandowski >Priority: Normal > > See the following ML threads: > * Branimir discussing having 2 supported configs which included a lot of > back and forth about what we're testing: > [link|https://lists.apache.org/thread/qf5c3hhz6qkpyqvbd3sppzlmftlc0bw0] > * Follow up DISCUSS thread notifying about this ticket: > [link|https://lists.apache.org/thread/r433j1mk111v102z4nr1m9l1fqf41fbr] > The pre-commit test suite has a lot of redundancy in our current approach > while also leaving certain configurations untested or undertested. We need to > address both sides of this. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-19421) expired data in sstable not remove
xxxcai999 created CASSANDRA-19421:
-
Summary: expired data in sstable not remove
Key: CASSANDRA-19421
URL: https://issues.apache.org/jira/browse/CASSANDRA-19421
Project: Cassandra
Issue Type: Bug
Reporter: xxxcai999
Attachments: sstabledump.txt
Version: cassandra:3.11.3
My table is using a TTL of 90 days. but sometimes expired data not deleted, The
file size will increase to several hundred gigabytes, occupying a large amount
of disk space.
Liked this sstable file:
-rw-r--r-- 1 cassandra cassandra 396G Sep 10 00:16 mc-22009-big-Data.db
-rw-r--r-- 1 cassandra cassandra 9 Sep 10 00:16 mc-22009-big-Digest.crc32
-rw-r--r-- 1 cassandra cassandra 37K Sep 10 00:16 mc-22009-big-Filter.db
-rw-r--r-- 1 cassandra cassandra 533M Sep 10 00:16 mc-22009-big-Index.db
-rw-r--r-- 1 cassandra cassandra 12K Sep 10 00:16 mc-22009-big-Statistics.db
-rw-r--r-- 1 cassandra cassandra 15K Sep 10 00:16 mc-22009-big-Summary.db
-rw-r--r-- 1 cassandra cassandra 92 Sep 10 00:16 mc-22009-big-TOC.txt
By using sstabledump tool, i found all rows expired
17 {
18 "type" : "row",
19 "position" : 104,
20 "clustering" : [ 1689491660981 ],
21 "liveness_info" : \{ "tstamp" : "2023-07-16T07:14:24.273004Z",
"ttl" : 7776000, "expires_at" : "2023-10-14T07:14:24Z", "expired" : true },
22 "cells" : [
23 \{ "name" : "long_v", "value" : 1 }
24 ]
25 }
This issue makes my disk space estimation unpredictable, so i want get some
help.
The attachment contains information on sstablemetadata.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
