[jira] [Updated] (CASSANDRA-19180) Support reloading certificate stores in cassandra-java-driver
[ https://issues.apache.org/jira/browse/CASSANDRA-19180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bret McGuire updated CASSANDRA-19180: - Reviewers: Andy Tolbert, Bret McGuire (was: absurdfarce#1, Andy Tolbert) Status: Review In Progress (was: Needs Committer) > Support reloading certificate stores in cassandra-java-driver > - > > Key: CASSANDRA-19180 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19180 > Project: Cassandra > Issue Type: New Feature > Components: Client/java-driver >Reporter: Abe Ratnofsky >Assignee: Abe Ratnofsky >Priority: Normal > Time Spent: 2h 50m > Remaining Estimate: 0h > > Currently, apache/cassandra-java-driver does not reload SSLContext when the > underlying certificate store files change. When the DefaultSslEngineFactory > (and the other factories) are set up, they build a fixed instance of > javax.net.ssl.SSLContext that doesn't change: > https://github.com/apache/cassandra-java-driver/blob/12e3e3ea027c51c5807e5e46ba542f894edfa4e7/core/src/main/java/com/datastax/oss/driver/internal/core/ssl/DefaultSslEngineFactory.java#L74 > This fixed SSLContext is used to negotiate SSL with the cluster, and if a > keystore is reloaded on disk it isn't picked up by the driver, and future > reconnections will fail if the keystore certificates have expired by the time > they're used to handshake a new connection. > We should reload client certificates so that applications that provide them > can use short-lived certificates and not require a bounce to pick up new > certificates. This is especially relevant in a world with CASSANDRA-18554 and > broad use of mTLS. > I have a patch for this that is nearly ready. Now that the project has moved > under apache/ - who can I work with to understand how CI works now? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19180) Support reloading certificate stores in cassandra-java-driver
[ https://issues.apache.org/jira/browse/CASSANDRA-19180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bret McGuire updated CASSANDRA-19180: - Status: Ready to Commit (was: Review In Progress) > Support reloading certificate stores in cassandra-java-driver > - > > Key: CASSANDRA-19180 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19180 > Project: Cassandra > Issue Type: New Feature > Components: Client/java-driver >Reporter: Abe Ratnofsky >Assignee: Abe Ratnofsky >Priority: Normal > Time Spent: 2h 50m > Remaining Estimate: 0h > > Currently, apache/cassandra-java-driver does not reload SSLContext when the > underlying certificate store files change. When the DefaultSslEngineFactory > (and the other factories) are set up, they build a fixed instance of > javax.net.ssl.SSLContext that doesn't change: > https://github.com/apache/cassandra-java-driver/blob/12e3e3ea027c51c5807e5e46ba542f894edfa4e7/core/src/main/java/com/datastax/oss/driver/internal/core/ssl/DefaultSslEngineFactory.java#L74 > This fixed SSLContext is used to negotiate SSL with the cluster, and if a > keystore is reloaded on disk it isn't picked up by the driver, and future > reconnections will fail if the keystore certificates have expired by the time > they're used to handshake a new connection. > We should reload client certificates so that applications that provide them > can use short-lived certificates and not require a bounce to pick up new > certificates. This is especially relevant in a world with CASSANDRA-18554 and > broad use of mTLS. > I have a patch for this that is nearly ready. Now that the project has moved > under apache/ - who can I work with to understand how CI works now? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18951) Add option for MutualTlsAuthenticator to restrict the certificate validity period
[
https://issues.apache.org/jira/browse/CASSANDRA-18951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francisco Guerrero updated CASSANDRA-18951:
---
Change Category: Operability (was: Semantic)
> Add option for MutualTlsAuthenticator to restrict the certificate validity
> period
> -
>
> Key: CASSANDRA-18951
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18951
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Authorization, Messaging/Client,
> Observability/JMX, Observability/Metrics
>Reporter: Francisco Guerrero
>Assignee: Francisco Guerrero
>Priority: Normal
> Fix For: 5.1
>
> Attachments: ci_summary.html, ci_summary.json, result_details.tar.gz
>
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> In {{org.apache.cassandra.auth.MutualTlsAuthenticator}}, we validate that a
> certificate is valid by looking at the identities inside the
> certificate and making sure the identity exists in the identity to role table.
> In some situations we may want to restrict the certificates
> we accept by rejecting certificates older than x amount of days. Some
> certificates can be generated with long expiration dates,
> and this might be undesired when you want to protect against potential
> certificates being compromised. For that reason, it is
> important to add an option, that when configured, we can limit the age of the
> certificate we accept for mTLS authentication.
> When enabled, this will force clients to have to renew certificates more
> frequently, reducing the exposure of a Cassandra cluster
> to leaked certificates.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18951) Add option for MutualTlsAuthenticator to restrict the certificate validity period
[
https://issues.apache.org/jira/browse/CASSANDRA-18951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francisco Guerrero updated CASSANDRA-18951:
---
Summary: Add option for MutualTlsAuthenticator to restrict the certificate
validity period (was: Add option for MutualTlsAuthenticator to restrict the
certificate age)
> Add option for MutualTlsAuthenticator to restrict the certificate validity
> period
> -
>
> Key: CASSANDRA-18951
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18951
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Authorization, Messaging/Client,
> Observability/JMX, Observability/Metrics
>Reporter: Francisco Guerrero
>Assignee: Francisco Guerrero
>Priority: Normal
> Fix For: 5.1
>
> Attachments: ci_summary.html, ci_summary.json, result_details.tar.gz
>
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> In {{org.apache.cassandra.auth.MutualTlsAuthenticator}}, we validate that a
> certificate is valid by looking at the identities inside the
> certificate and making sure the identity exists in the identity to role table.
> In some situations we may want to restrict the certificates
> we accept by rejecting certificates older than x amount of days. Some
> certificates can be generated with long expiration dates,
> and this might be undesired when you want to protect against potential
> certificates being compromised. For that reason, it is
> important to add an option, that when configured, we can limit the age of the
> certificate we accept for mTLS authentication.
> When enabled, this will force clients to have to renew certificates more
> frequently, reducing the exposure of a Cassandra cluster
> to leaked certificates.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18951) Add option for MutualTlsAuthenticator to restrict the certificate age
[
https://issues.apache.org/jira/browse/CASSANDRA-18951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francisco Guerrero updated CASSANDRA-18951:
---
Component/s: Messaging/Client
Observability/JMX
Observability/Metrics
> Add option for MutualTlsAuthenticator to restrict the certificate age
> -
>
> Key: CASSANDRA-18951
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18951
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Authorization, Messaging/Client,
> Observability/JMX, Observability/Metrics
>Reporter: Francisco Guerrero
>Assignee: Francisco Guerrero
>Priority: Normal
> Fix For: 5.1
>
> Attachments: ci_summary.html, ci_summary.json, result_details.tar.gz
>
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> In {{org.apache.cassandra.auth.MutualTlsAuthenticator}}, we validate that a
> certificate is valid by looking at the identities inside the
> certificate and making sure the identity exists in the identity to role table.
> In some situations we may want to restrict the certificates
> we accept by rejecting certificates older than x amount of days. Some
> certificates can be generated with long expiration dates,
> and this might be undesired when you want to protect against potential
> certificates being compromised. For that reason, it is
> important to add an option, that when configured, we can limit the age of the
> certificate we accept for mTLS authentication.
> When enabled, this will force clients to have to renew certificates more
> frequently, reducing the exposure of a Cassandra cluster
> to leaked certificates.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18951) Add option for MutualTlsAuthenticator to restrict the certificate age
[
https://issues.apache.org/jira/browse/CASSANDRA-18951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francisco Guerrero updated CASSANDRA-18951:
---
Fix Version/s: 5.1
Source Control Link:
https://github.com/apache/cassandra/commit/a0af41f666c23a840d9df3f06729ed5fd2c06cd1
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
> Add option for MutualTlsAuthenticator to restrict the certificate age
> -
>
> Key: CASSANDRA-18951
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18951
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Authorization
>Reporter: Francisco Guerrero
>Assignee: Francisco Guerrero
>Priority: Normal
> Fix For: 5.1
>
> Attachments: ci_summary.html, ci_summary.json, result_details.tar.gz
>
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> In {{org.apache.cassandra.auth.MutualTlsAuthenticator}}, we validate that a
> certificate is valid by looking at the identities inside the
> certificate and making sure the identity exists in the identity to role table.
> In some situations we may want to restrict the certificates
> we accept by rejecting certificates older than x amount of days. Some
> certificates can be generated with long expiration dates,
> and this might be undesired when you want to protect against potential
> certificates being compromised. For that reason, it is
> important to add an option, that when configured, we can limit the age of the
> certificate we accept for mTLS authentication.
> When enabled, this will force clients to have to renew certificates more
> frequently, reducing the exposure of a Cassandra cluster
> to leaked certificates.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18951) Add option for MutualTlsAuthenticator to restrict the certificate age
[
https://issues.apache.org/jira/browse/CASSANDRA-18951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830735#comment-17830735
]
Francisco Guerrero commented on CASSANDRA-18951:
Attached green CI results [^ci_summary.html] [^ci_summary.json]
[^result_details.tar.gz]
> Add option for MutualTlsAuthenticator to restrict the certificate age
> -
>
> Key: CASSANDRA-18951
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18951
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Authorization
>Reporter: Francisco Guerrero
>Assignee: Francisco Guerrero
>Priority: Normal
> Attachments: ci_summary.html, ci_summary.json, result_details.tar.gz
>
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> In {{org.apache.cassandra.auth.MutualTlsAuthenticator}}, we validate that a
> certificate is valid by looking at the identities inside the
> certificate and making sure the identity exists in the identity to role table.
> In some situations we may want to restrict the certificates
> we accept by rejecting certificates older than x amount of days. Some
> certificates can be generated with long expiration dates,
> and this might be undesired when you want to protect against potential
> certificates being compromised. For that reason, it is
> important to add an option, that when configured, we can limit the age of the
> certificate we accept for mTLS authentication.
> When enabled, this will force clients to have to renew certificates more
> frequently, reducing the exposure of a Cassandra cluster
> to leaked certificates.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch trunk updated: CASSANDRA-18951: Add option for MutualTlsAuthenticator to restrict the certificate validity period
This is an automated email from the ASF dual-hosted git repository. frankgh pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git The following commit(s) were added to refs/heads/trunk by this push: new a0af41f666 CASSANDRA-18951: Add option for MutualTlsAuthenticator to restrict the certificate validity period a0af41f666 is described below commit a0af41f666c23a840d9df3f06729ed5fd2c06cd1 Author: Francisco Guerrero AuthorDate: Thu Feb 15 13:19:28 2024 -0800 CASSANDRA-18951: Add option for MutualTlsAuthenticator to restrict the certificate validity period In this commit, we introduce two new optional options for the `server_encryption_options` and the `client_encryption_options`. The options are `max_certificate_validity_period` and `certificate_validity_warn_threshold`. Both options can be configured as a duration configuration parameter as defined by the `DurationSpec` (see CASSANDRA-15234). The resolution for these new properties is minutes. When specified, the certificate validation implementation will take that information and reject certificates that are older than the maximum allowed certificate validity period, translating into a rejection from the authenticating user. The `certificate_validity_warn_threshold` option can be configured to emit warnings (log entries) when the certificate exceeds the validity threshold. patch by Francisco Guerrero; reviewed by Andy Tolbert, Abe Ratnofsky, Dinesh Joshi for CASSANDRA-18951 --- conf/cassandra.yaml| 16 + .../cassandra/auth/MutualTlsAuthenticator.java | 45 ++- .../auth/MutualTlsCertificateValidator.java| 27 +- ...utualTlsCertificateValidityPeriodValidator.java | 85 + .../auth/MutualTlsInternodeAuthenticator.java | 46 ++- .../org/apache/cassandra/auth/MutualTlsUtil.java | 83 + .../cassandra/auth/SpiffeCertificateValidator.java | 35 +- .../apache/cassandra/config/EncryptionOptions.java | 179 --- .../apache/cassandra/metrics/MutualTlsMetrics.java | 50 +++ test/data/config/version=5.0-alpha1.yml| 2 + .../cassandra/distributed/shared/ClusterUtils.java | 30 ++ .../distributed/test/JavaDriverUtils.java | 29 +- .../MutualTlsCertificateValidityPeriodTest.java| 351 + ...lTlsCertificateValidityPeriodValidatorTest.java | 86 + .../apache/cassandra/auth/MutualTlsUtilTest.java | 51 +++ .../auth/SpiffeCertificateValidatorTest.java | 2 +- .../cassandra/config/EncryptionOptionsTest.java| 53 +++- .../cassandra/utils/tls/CertificateBuilder.java| 234 ++ .../cassandra/utils/tls/CertificateBundle.java | 109 +++ 19 files changed, 1412 insertions(+), 101 deletions(-) diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml index a9393d4d18..42f311b0eb 100644 --- a/conf/cassandra.yaml +++ b/conf/cassandra.yaml @@ -1596,6 +1596,13 @@ server_encryption_options: # TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, # TLS_RSA_WITH_AES_256_CBC_SHA # ] + # Optional setting to define the maximum allowed validity period of the client certificate used for the internode + # inbound connections. For example, if the specified max_certificate_validity_period is 30 days and the client + # uses a certificate that is issued for more than 30 days, the connection will be rejected. + # max_certificate_validity_period: 365d + # Optional setting that defines a warning threshold. When the threshold is exceeded for the internode certificate + # validity period, warnings with information about the certificate expiration will be logged. + # certificate_validity_warn_threshold: 10d # Configure client-to-server encryption. # @@ -1642,6 +1649,15 @@ client_encryption_options: # TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, # TLS_RSA_WITH_AES_256_CBC_SHA # ] + # Optional setting to define the maximum validity period of the client certificate allowed to establish + # connections to the server. For example, if max_certificate_validity_period is configured for 10 days, + # and a client attempts to authenticate with a certificate with a longer validity period (say 30 days), + # then the connection will be rejected. + # max_certificate_validity_period: 365d + # Optional setting that defines a warning threshold. When the threshold is exceeded for the client certificate + # validity, warnings with information about the certificate expiration will be logged. Additionally, client + # warnings will be reported during the session establishment. + # certificate_validity_warn_threshold: 10d # internode_compression controls whether traffic between nodes is # compressed. diff --git a/src/java/org/apache/cassandra/auth/MutualTlsAuthenticator.java b/src/jav
[jira] [Updated] (CASSANDRA-18951) Add option for MutualTlsAuthenticator to restrict the certificate age
[
https://issues.apache.org/jira/browse/CASSANDRA-18951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francisco Guerrero updated CASSANDRA-18951:
---
Attachment: ci_summary.html
ci_summary.json
result_details.tar.gz
> Add option for MutualTlsAuthenticator to restrict the certificate age
> -
>
> Key: CASSANDRA-18951
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18951
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Authorization
>Reporter: Francisco Guerrero
>Assignee: Francisco Guerrero
>Priority: Normal
> Attachments: ci_summary.html, ci_summary.json, result_details.tar.gz
>
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> In {{org.apache.cassandra.auth.MutualTlsAuthenticator}}, we validate that a
> certificate is valid by looking at the identities inside the
> certificate and making sure the identity exists in the identity to role table.
> In some situations we may want to restrict the certificates
> we accept by rejecting certificates older than x amount of days. Some
> certificates can be generated with long expiration dates,
> and this might be undesired when you want to protect against potential
> certificates being compromised. For that reason, it is
> important to add an option, that when configured, we can limit the age of the
> certificate we accept for mTLS authentication.
> When enabled, this will force clients to have to renew certificates more
> frequently, reducing the exposure of a Cassandra cluster
> to leaked certificates.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra-website) branch asf-staging updated (4b2dd127f -> e496d3a63)
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a change to branch asf-staging in repository https://gitbox.apache.org/repos/asf/cassandra-website.git discard 4b2dd127f generate docs for fd550e9c add 85c413711 Update keyserver used on downloads page from pool.sks-keyservers.net to keyserver.ubuntu.com new e496d3a63 generate docs for 85c41371 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (4b2dd127f) \ N -- N -- N refs/heads/asf-staging (e496d3a63) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: content/_/download.html| 2 +- content/search-index.js| 2 +- .../source/modules/ROOT/pages/download.adoc| 2 +- site-ui/build/ui-bundle.zip| Bin 4883646 -> 4883646 bytes 4 files changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra-website) branch trunk updated: Update keyserver used on downloads page from pool.sks-keyservers.net to keyserver.ubuntu.com
This is an automated email from the ASF dual-hosted git repository. mck pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra-website.git The following commit(s) were added to refs/heads/trunk by this push: new 85c413711 Update keyserver used on downloads page from pool.sks-keyservers.net to keyserver.ubuntu.com 85c413711 is described below commit 85c41371156af66c49938223dffcec2418143910 Author: mck AuthorDate: Mon Mar 25 23:49:46 2024 +0100 Update keyserver used on downloads page from pool.sks-keyservers.net to keyserver.ubuntu.com ref: https://the-asf.slack.com/archives/CJZLTM05A/p1711357217492399 patch by Mick Semb Wever; reviewed by Simon Karberg --- site-content/source/modules/ROOT/pages/download.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site-content/source/modules/ROOT/pages/download.adoc b/site-content/source/modules/ROOT/pages/download.adoc index 17ebab8fb..1010e9096 100644 --- a/site-content/source/modules/ROOT/pages/download.adoc +++ b/site-content/source/modules/ROOT/pages/download.adoc @@ -188,7 +188,7 @@ Then add the public key A278B781FE4B2BDA as follows: [source] -- -sudo apt-key adv --keyserver pool.sks-keyservers.net --recv-key A278B781FE4B2BDA +sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-key A278B781FE4B2BDA -- and repeat `sudo apt-get update`. The actual key may be different, you get it from the error message itself. For a full list of Apache contributors public keys, you can refer to https://downloads.apache.org/cassandra/KEYS[Cassandra KEYS]. - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18951) Add option for MutualTlsAuthenticator to restrict the certificate age
[
https://issues.apache.org/jira/browse/CASSANDRA-18951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francisco Guerrero updated CASSANDRA-18951:
---
Reviewers: Abe Ratnofsky, Andy Tolbert, Dinesh Joshi (was: Abe Ratnofsky,
Andy Tolbert)
> Add option for MutualTlsAuthenticator to restrict the certificate age
> -
>
> Key: CASSANDRA-18951
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18951
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Authorization
>Reporter: Francisco Guerrero
>Assignee: Francisco Guerrero
>Priority: Normal
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> In {{org.apache.cassandra.auth.MutualTlsAuthenticator}}, we validate that a
> certificate is valid by looking at the identities inside the
> certificate and making sure the identity exists in the identity to role table.
> In some situations we may want to restrict the certificates
> we accept by rejecting certificates older than x amount of days. Some
> certificates can be generated with long expiration dates,
> and this might be undesired when you want to protect against potential
> certificates being compromised. For that reason, it is
> important to add an option, that when configured, we can limit the age of the
> certificate we accept for mTLS authentication.
> When enabled, this will force clients to have to renew certificates more
> frequently, reducing the exposure of a Cassandra cluster
> to leaked certificates.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18951) Add option for MutualTlsAuthenticator to restrict the certificate age
[
https://issues.apache.org/jira/browse/CASSANDRA-18951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francisco Guerrero updated CASSANDRA-18951:
---
Status: Ready to Commit (was: Review In Progress)
I will be rebasing once again, and I will upload test results right after that.
> Add option for MutualTlsAuthenticator to restrict the certificate age
> -
>
> Key: CASSANDRA-18951
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18951
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Authorization
>Reporter: Francisco Guerrero
>Assignee: Francisco Guerrero
>Priority: Normal
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> In {{org.apache.cassandra.auth.MutualTlsAuthenticator}}, we validate that a
> certificate is valid by looking at the identities inside the
> certificate and making sure the identity exists in the identity to role table.
> In some situations we may want to restrict the certificates
> we accept by rejecting certificates older than x amount of days. Some
> certificates can be generated with long expiration dates,
> and this might be undesired when you want to protect against potential
> certificates being compromised. For that reason, it is
> important to add an option, that when configured, we can limit the age of the
> certificate we accept for mTLS authentication.
> When enabled, this will force clients to have to renew certificates more
> frequently, reducing the exposure of a Cassandra cluster
> to leaked certificates.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
Re: [PR] Rack-aware weighted load balancing policy [cassandra-java-driver]
aratno commented on code in PR #1922:
URL:
https://github.com/apache/cassandra-java-driver/pull/1922#discussion_r1538206832
##
core/src/main/java/com/datastax/oss/driver/internal/core/loadbalancing/RackAwareWeightedLoadBalancingPolicy.java:
##
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.datastax.oss.driver.internal.core.loadbalancing;
+
+import static java.util.concurrent.TimeUnit.SECONDS;
+
+import com.datastax.oss.driver.api.core.config.DefaultDriverOption;
+import com.datastax.oss.driver.api.core.context.DriverContext;
+import com.datastax.oss.driver.api.core.metadata.Node;
+import com.datastax.oss.driver.api.core.session.Request;
+import com.datastax.oss.driver.api.core.session.Session;
+import com.datastax.oss.driver.internal.core.util.ArrayUtils;
+import com.datastax.oss.driver.internal.core.util.collection.QueryPlan;
+import com.datastax.oss.driver.internal.core.util.collection.SimpleQueryPlan;
+import edu.umd.cs.findbugs.annotations.NonNull;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Objects;
+import java.util.Queue;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A load balancing policy that optimally balances between sending load to
local token holder,
+ * rack replicas, and local datacenter replicas (in that order).
+ *
+ * The default weights are good for the vast majority of use cases, but you
can tweak them to get different behavior.
+ */
+public class RackAwareWeightedLoadBalancingPolicy extends
DefaultLoadBalancingPolicy {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RackAwareWeightedLoadBalancingPolicy.class);
+ // Each client will randomly skew so traffic is introduced gradually to a
newly up replica
+ // Each client will start sending at a period between 15s and 30, and they
will gradually
+ // increase load over the next 15 seconds.
+ private static final long DELAY_TRAFFIC_SKEW_MILLIS = SECONDS.toMillis(15);
+ private static final long DELAY_TRAFFIC_MILLIS =
+ DELAY_TRAFFIC_SKEW_MILLIS +
ThreadLocalRandom.current().nextLong(DELAY_TRAFFIC_SKEW_MILLIS);
+
+ // By default we will only score this many nodes, the rest will get added on
without scoring.
+ // We don't usually need to score every single node if there are more than a
few.
+ static final int DEFAULT_SCORED_PLAN_SIZE = 8;
+ // Default multiplicative weights. Think of this like "How much concurrency
must occur
+ // before I fail off this node to the next". Note that these defaults are
intentionally
+ // meant to shed load to unloaded rack coordinators when a replica set is all
+ // relatively heavily loaded (specifically 3x as loaded).
+ static final double DEFAULT_WEIGHT_NON_RACK = 4.0;
+ static final double DEFAULT_WEIGHT_NON_REPLICA = 12.0;
+ static final double DEFAULT_WEIGHT_STARTING = 16.0;
+ static final double DEFAULT_WEIGHT_UNHEALTHY = 64.0;
+
+ private final int planSize;
+ private final double weightNonRack;
+ private final double weightNonReplica;
+ private final double weightStarting;
+ private final double weightUnhealthy;
+
+ public RackAwareWeightedLoadBalancingPolicy(
+ @NonNull DriverContext context,
+ @NonNull String profileName) {
+super(context, profileName);
+this.planSize =
profile.getInt(DefaultDriverOption.LOAD_BALANCING_SCORED_PLAN_SIZE,
DEFAULT_SCORED_PLAN_SIZE);
+// Choices of weights will change how this load balancer prefers endpoints.
+// The weight is relative to the outstanding concurrency.
+this.weightNonRack = profile.getDouble(
+DefaultDriverOption.LOAD_BALANCING_WEIGHT_NON_RACK,
DEFAULT_WEIGHT_NON_RACK);
+this.weightNonReplica = profile.getDouble(
+DefaultDriverOption.LOAD_BALANCING_WEIGHT_NON_REPLICA,
DEFAULT_WEIGHT_NON_REPLICA);
+this.weightStarting = profile.getDouble(
+DefaultDriverOption.LOAD_BALANCING_WEIGHT_STARTING,
DEFAULT_WEIGHT_STARTING);
+this.weightUnhealthy = profile.getDouble(
+DefaultDriverOption.LOAD_BALANCING_WEIGHT_UNHEALTHY,
DEFAULT_WEIGHT_UNHEALTHY);
+ }
+
+ @NonNull
+
[jira] [Updated] (CASSANDRA-19341) Relation and Restriction hierachies are too complex and error prone
[
https://issues.apache.org/jira/browse/CASSANDRA-19341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ekaterina Dimitrova updated CASSANDRA-19341:
Reviewers: Ekaterina Dimitrova
Status: Review In Progress (was: Patch Available)
> Relation and Restriction hierachies are too complex and error prone
> ---
>
> Key: CASSANDRA-19341
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19341
> Project: Cassandra
> Issue Type: Improvement
> Components: CQL/Interpreter
>Reporter: Benjamin Lerer
>Assignee: Benjamin Lerer
>Priority: Normal
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The {{Relation}} and {{Restriction}} hierarchy have been designed when C* was
> only supporting a limited amount of operators and columns expressions (single
> column, multi-column and token expressions). Over time they have grown in
> complexity making the code harder to understand and modify and error prone.
> Their design is also resulting in unnecessary limitations that could be
> easily lifted, like the ability to accept different predicates on the same
> column.
> Today adding a new operator requires the addition of a lot of glue code and
> surgical changes accross the CQL layer. Making patch for features such as
> CASSANDRA-18584 much complex than it should be.
> The goal of this ticket is to simplify the {{Relation}} and {{Restriction}}
> hierarchies and modify operator class so that adding new operators requires
> only changes to the {{Operator}} class and ANTLR file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
Re: [PR] Rack-aware weighted load balancing policy [cassandra-java-driver]
akhaku commented on code in PR #1922:
URL:
https://github.com/apache/cassandra-java-driver/pull/1922#discussion_r1538193530
##
core/src/main/java/com/datastax/oss/driver/internal/core/loadbalancing/RackAwareWeightedLoadBalancingPolicy.java:
##
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.datastax.oss.driver.internal.core.loadbalancing;
+
+import static java.util.concurrent.TimeUnit.SECONDS;
+
+import com.datastax.oss.driver.api.core.config.DefaultDriverOption;
+import com.datastax.oss.driver.api.core.context.DriverContext;
+import com.datastax.oss.driver.api.core.metadata.Node;
+import com.datastax.oss.driver.api.core.session.Request;
+import com.datastax.oss.driver.api.core.session.Session;
+import com.datastax.oss.driver.internal.core.util.ArrayUtils;
+import com.datastax.oss.driver.internal.core.util.collection.QueryPlan;
+import com.datastax.oss.driver.internal.core.util.collection.SimpleQueryPlan;
+import edu.umd.cs.findbugs.annotations.NonNull;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Objects;
+import java.util.Queue;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A load balancing policy that optimally balances between sending load to
local token holder,
+ * rack replicas, and local datacenter replicas (in that order).
+ *
+ * The default weights are good for the vast majority of use cases, but you
can tweak them to get different behavior.
+ */
+public class RackAwareWeightedLoadBalancingPolicy extends
DefaultLoadBalancingPolicy {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RackAwareWeightedLoadBalancingPolicy.class);
+ // Each client will randomly skew so traffic is introduced gradually to a
newly up replica
+ // Each client will start sending at a period between 15s and 30, and they
will gradually
+ // increase load over the next 15 seconds.
+ private static final long DELAY_TRAFFIC_SKEW_MILLIS = SECONDS.toMillis(15);
+ private static final long DELAY_TRAFFIC_MILLIS =
+ DELAY_TRAFFIC_SKEW_MILLIS +
ThreadLocalRandom.current().nextLong(DELAY_TRAFFIC_SKEW_MILLIS);
+
+ // By default we will only score this many nodes, the rest will get added on
without scoring.
+ // We don't usually need to score every single node if there are more than a
few.
+ static final int DEFAULT_SCORED_PLAN_SIZE = 8;
+ // Default multiplicative weights. Think of this like "How much concurrency
must occur
+ // before I fail off this node to the next". Note that these defaults are
intentionally
+ // meant to shed load to unloaded rack coordinators when a replica set is all
+ // relatively heavily loaded (specifically 3x as loaded).
+ static final double DEFAULT_WEIGHT_NON_RACK = 4.0;
+ static final double DEFAULT_WEIGHT_NON_REPLICA = 12.0;
+ static final double DEFAULT_WEIGHT_STARTING = 16.0;
+ static final double DEFAULT_WEIGHT_UNHEALTHY = 64.0;
+
+ private final int planSize;
+ private final double weightNonRack;
+ private final double weightNonReplica;
+ private final double weightStarting;
+ private final double weightUnhealthy;
+
+ public RackAwareWeightedLoadBalancingPolicy(
+ @NonNull DriverContext context,
+ @NonNull String profileName) {
+super(context, profileName);
+this.planSize =
profile.getInt(DefaultDriverOption.LOAD_BALANCING_SCORED_PLAN_SIZE,
DEFAULT_SCORED_PLAN_SIZE);
+// Choices of weights will change how this load balancer prefers endpoints.
+// The weight is relative to the outstanding concurrency.
+this.weightNonRack = profile.getDouble(
+DefaultDriverOption.LOAD_BALANCING_WEIGHT_NON_RACK,
DEFAULT_WEIGHT_NON_RACK);
+this.weightNonReplica = profile.getDouble(
+DefaultDriverOption.LOAD_BALANCING_WEIGHT_NON_REPLICA,
DEFAULT_WEIGHT_NON_REPLICA);
+this.weightStarting = profile.getDouble(
+DefaultDriverOption.LOAD_BALANCING_WEIGHT_STARTING,
DEFAULT_WEIGHT_STARTING);
+this.weightUnhealthy = profile.getDouble(
+DefaultDriverOption.LOAD_BALANCING_WEIGHT_UNHEALTHY,
DEFAULT_WEIGHT_UNHEALTHY);
+ }
+
+ @NonNull
+
Re: [PR] Rack-aware weighted load balancing policy [cassandra-java-driver]
aratno commented on code in PR #1922:
URL:
https://github.com/apache/cassandra-java-driver/pull/1922#discussion_r1538189365
##
core/src/main/java/com/datastax/oss/driver/internal/core/loadbalancing/RackAwareWeightedLoadBalancingPolicy.java:
##
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.datastax.oss.driver.internal.core.loadbalancing;
+
+import static java.util.concurrent.TimeUnit.SECONDS;
+
+import com.datastax.oss.driver.api.core.config.DefaultDriverOption;
+import com.datastax.oss.driver.api.core.context.DriverContext;
+import com.datastax.oss.driver.api.core.metadata.Node;
+import com.datastax.oss.driver.api.core.session.Request;
+import com.datastax.oss.driver.api.core.session.Session;
+import com.datastax.oss.driver.internal.core.util.ArrayUtils;
+import com.datastax.oss.driver.internal.core.util.collection.QueryPlan;
+import com.datastax.oss.driver.internal.core.util.collection.SimpleQueryPlan;
+import edu.umd.cs.findbugs.annotations.NonNull;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Objects;
+import java.util.Queue;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A load balancing policy that optimally balances between sending load to
local token holder,
+ * rack replicas, and local datacenter replicas (in that order).
+ *
+ * The default weights are good for the vast majority of use cases, but you
can tweak them to get different behavior.
+ */
+public class RackAwareWeightedLoadBalancingPolicy extends
DefaultLoadBalancingPolicy {
Review Comment:
Yeah, I think the changes outside of rack-awareness make this worth
recommending broadly. I wouldn't support making it the default until it's been
available for at least a release, but worth pushing it in that direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18951) Add option for MutualTlsAuthenticator to restrict the certificate age
[
https://issues.apache.org/jira/browse/CASSANDRA-18951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830690#comment-17830690
]
Dinesh Joshi commented on CASSANDRA-18951:
--
+1, thanks for the patch.
> Add option for MutualTlsAuthenticator to restrict the certificate age
> -
>
> Key: CASSANDRA-18951
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18951
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Authorization
>Reporter: Francisco Guerrero
>Assignee: Francisco Guerrero
>Priority: Normal
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> In {{org.apache.cassandra.auth.MutualTlsAuthenticator}}, we validate that a
> certificate is valid by looking at the identities inside the
> certificate and making sure the identity exists in the identity to role table.
> In some situations we may want to restrict the certificates
> we accept by rejecting certificates older than x amount of days. Some
> certificates can be generated with long expiration dates,
> and this might be undesired when you want to protect against potential
> certificates being compromised. For that reason, it is
> important to add an option, that when configured, we can limit the age of the
> certificate we accept for mTLS authentication.
> When enabled, this will force clients to have to renew certificates more
> frequently, reducing the exposure of a Cassandra cluster
> to leaked certificates.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
Re: [PR] Rack-aware weighted load balancing policy [cassandra-java-driver]
akhaku commented on code in PR #1922:
URL:
https://github.com/apache/cassandra-java-driver/pull/1922#discussion_r1538180527
##
core/src/main/java/com/datastax/oss/driver/internal/core/loadbalancing/RackAwareWeightedLoadBalancingPolicy.java:
##
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.datastax.oss.driver.internal.core.loadbalancing;
+
+import static java.util.concurrent.TimeUnit.SECONDS;
+
+import com.datastax.oss.driver.api.core.config.DefaultDriverOption;
+import com.datastax.oss.driver.api.core.context.DriverContext;
+import com.datastax.oss.driver.api.core.metadata.Node;
+import com.datastax.oss.driver.api.core.session.Request;
+import com.datastax.oss.driver.api.core.session.Session;
+import com.datastax.oss.driver.internal.core.util.ArrayUtils;
+import com.datastax.oss.driver.internal.core.util.collection.QueryPlan;
+import com.datastax.oss.driver.internal.core.util.collection.SimpleQueryPlan;
+import edu.umd.cs.findbugs.annotations.NonNull;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Objects;
+import java.util.Queue;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A load balancing policy that optimally balances between sending load to
local token holder,
+ * rack replicas, and local datacenter replicas (in that order).
+ *
+ * The default weights are good for the vast majority of use cases, but you
can tweak them to get different behavior.
+ */
+public class RackAwareWeightedLoadBalancingPolicy extends
DefaultLoadBalancingPolicy {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RackAwareWeightedLoadBalancingPolicy.class);
+ // Each client will randomly skew so traffic is introduced gradually to a
newly up replica
+ // Each client will start sending at a period between 15s and 30, and they
will gradually
+ // increase load over the next 15 seconds.
+ private static final long DELAY_TRAFFIC_SKEW_MILLIS = SECONDS.toMillis(15);
+ private static final long DELAY_TRAFFIC_MILLIS =
+ DELAY_TRAFFIC_SKEW_MILLIS +
ThreadLocalRandom.current().nextLong(DELAY_TRAFFIC_SKEW_MILLIS);
+
+ // By default we will only score this many nodes, the rest will get added on
without scoring.
+ // We don't usually need to score every single node if there are more than a
few.
+ static final int DEFAULT_SCORED_PLAN_SIZE = 8;
+ // Default multiplicative weights. Think of this like "How much concurrency
must occur
+ // before I fail off this node to the next". Note that these defaults are
intentionally
+ // meant to shed load to unloaded rack coordinators when a replica set is all
+ // relatively heavily loaded (specifically 3x as loaded).
+ static final double DEFAULT_WEIGHT_NON_RACK = 4.0;
+ static final double DEFAULT_WEIGHT_NON_REPLICA = 12.0;
+ static final double DEFAULT_WEIGHT_STARTING = 16.0;
+ static final double DEFAULT_WEIGHT_UNHEALTHY = 64.0;
+
+ private final int planSize;
+ private final double weightNonRack;
+ private final double weightNonReplica;
+ private final double weightStarting;
+ private final double weightUnhealthy;
+
+ public RackAwareWeightedLoadBalancingPolicy(
+ @NonNull DriverContext context,
+ @NonNull String profileName) {
+super(context, profileName);
+this.planSize =
profile.getInt(DefaultDriverOption.LOAD_BALANCING_SCORED_PLAN_SIZE,
DEFAULT_SCORED_PLAN_SIZE);
+// Choices of weights will change how this load balancer prefers endpoints.
+// The weight is relative to the outstanding concurrency.
+this.weightNonRack = profile.getDouble(
+DefaultDriverOption.LOAD_BALANCING_WEIGHT_NON_RACK,
DEFAULT_WEIGHT_NON_RACK);
+this.weightNonReplica = profile.getDouble(
+DefaultDriverOption.LOAD_BALANCING_WEIGHT_NON_REPLICA,
DEFAULT_WEIGHT_NON_REPLICA);
+this.weightStarting = profile.getDouble(
+DefaultDriverOption.LOAD_BALANCING_WEIGHT_STARTING,
DEFAULT_WEIGHT_STARTING);
+this.weightUnhealthy = profile.getDouble(
+DefaultDriverOption.LOAD_BALANCING_WEIGHT_UNHEALTHY,
DEFAULT_WEIGHT_UNHEALTHY);
+ }
+
+ @NonNull
+
Re: [PR] Rack-aware weighted load balancing policy [cassandra-java-driver]
aratno commented on code in PR #1922:
URL:
https://github.com/apache/cassandra-java-driver/pull/1922#discussion_r1538174008
##
core/src/main/java/com/datastax/oss/driver/internal/core/loadbalancing/RackAwareWeightedLoadBalancingPolicy.java:
##
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.datastax.oss.driver.internal.core.loadbalancing;
+
+import static java.util.concurrent.TimeUnit.SECONDS;
+
+import com.datastax.oss.driver.api.core.config.DefaultDriverOption;
+import com.datastax.oss.driver.api.core.context.DriverContext;
+import com.datastax.oss.driver.api.core.metadata.Node;
+import com.datastax.oss.driver.api.core.session.Request;
+import com.datastax.oss.driver.api.core.session.Session;
+import com.datastax.oss.driver.internal.core.util.ArrayUtils;
+import com.datastax.oss.driver.internal.core.util.collection.QueryPlan;
+import com.datastax.oss.driver.internal.core.util.collection.SimpleQueryPlan;
+import edu.umd.cs.findbugs.annotations.NonNull;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Objects;
+import java.util.Queue;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A load balancing policy that optimally balances between sending load to
local token holder,
+ * rack replicas, and local datacenter replicas (in that order).
+ *
+ * The default weights are good for the vast majority of use cases, but you
can tweak them to get different behavior.
+ */
+public class RackAwareWeightedLoadBalancingPolicy extends
DefaultLoadBalancingPolicy {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RackAwareWeightedLoadBalancingPolicy.class);
+ // Each client will randomly skew so traffic is introduced gradually to a
newly up replica
+ // Each client will start sending at a period between 15s and 30, and they
will gradually
+ // increase load over the next 15 seconds.
+ private static final long DELAY_TRAFFIC_SKEW_MILLIS = SECONDS.toMillis(15);
+ private static final long DELAY_TRAFFIC_MILLIS =
+ DELAY_TRAFFIC_SKEW_MILLIS +
ThreadLocalRandom.current().nextLong(DELAY_TRAFFIC_SKEW_MILLIS);
+
+ // By default we will only score this many nodes, the rest will get added on
without scoring.
+ // We don't usually need to score every single node if there are more than a
few.
+ static final int DEFAULT_SCORED_PLAN_SIZE = 8;
Review Comment:
It's a reasonable default - just thinking through what we should document
for tuning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
Re: [PR] Rack-aware weighted load balancing policy [cassandra-java-driver]
akhaku commented on code in PR #1922:
URL:
https://github.com/apache/cassandra-java-driver/pull/1922#discussion_r1538173041
##
core/src/main/java/com/datastax/oss/driver/internal/core/loadbalancing/RackAwareWeightedLoadBalancingPolicy.java:
##
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.datastax.oss.driver.internal.core.loadbalancing;
+
+import static java.util.concurrent.TimeUnit.SECONDS;
+
+import com.datastax.oss.driver.api.core.config.DefaultDriverOption;
+import com.datastax.oss.driver.api.core.context.DriverContext;
+import com.datastax.oss.driver.api.core.metadata.Node;
+import com.datastax.oss.driver.api.core.session.Request;
+import com.datastax.oss.driver.api.core.session.Session;
+import com.datastax.oss.driver.internal.core.util.ArrayUtils;
+import com.datastax.oss.driver.internal.core.util.collection.QueryPlan;
+import com.datastax.oss.driver.internal.core.util.collection.SimpleQueryPlan;
+import edu.umd.cs.findbugs.annotations.NonNull;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Objects;
+import java.util.Queue;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A load balancing policy that optimally balances between sending load to
local token holder,
+ * rack replicas, and local datacenter replicas (in that order).
+ *
+ * The default weights are good for the vast majority of use cases, but you
can tweak them to get different behavior.
+ */
+public class RackAwareWeightedLoadBalancingPolicy extends
DefaultLoadBalancingPolicy {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RackAwareWeightedLoadBalancingPolicy.class);
+ // Each client will randomly skew so traffic is introduced gradually to a
newly up replica
+ // Each client will start sending at a period between 15s and 30, and they
will gradually
+ // increase load over the next 15 seconds.
+ private static final long DELAY_TRAFFIC_SKEW_MILLIS = SECONDS.toMillis(15);
+ private static final long DELAY_TRAFFIC_MILLIS =
+ DELAY_TRAFFIC_SKEW_MILLIS +
ThreadLocalRandom.current().nextLong(DELAY_TRAFFIC_SKEW_MILLIS);
+
+ // By default we will only score this many nodes, the rest will get added on
without scoring.
+ // We don't usually need to score every single node if there are more than a
few.
+ static final int DEFAULT_SCORED_PLAN_SIZE = 8;
Review Comment:
That's fair, this default was chosen to fit with our requirements but a
larger default may make sense for a more general use case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
Re: [PR] Rack-aware weighted load balancing policy [cassandra-java-driver]
akhaku commented on code in PR #1922:
URL:
https://github.com/apache/cassandra-java-driver/pull/1922#discussion_r1538171418
##
core/src/main/java/com/datastax/oss/driver/internal/core/loadbalancing/RackAwareWeightedLoadBalancingPolicy.java:
##
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.datastax.oss.driver.internal.core.loadbalancing;
+
+import static java.util.concurrent.TimeUnit.SECONDS;
+
+import com.datastax.oss.driver.api.core.config.DefaultDriverOption;
+import com.datastax.oss.driver.api.core.context.DriverContext;
+import com.datastax.oss.driver.api.core.metadata.Node;
+import com.datastax.oss.driver.api.core.session.Request;
+import com.datastax.oss.driver.api.core.session.Session;
+import com.datastax.oss.driver.internal.core.util.ArrayUtils;
+import com.datastax.oss.driver.internal.core.util.collection.QueryPlan;
+import com.datastax.oss.driver.internal.core.util.collection.SimpleQueryPlan;
+import edu.umd.cs.findbugs.annotations.NonNull;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Objects;
+import java.util.Queue;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A load balancing policy that optimally balances between sending load to
local token holder,
+ * rack replicas, and local datacenter replicas (in that order).
+ *
+ * The default weights are good for the vast majority of use cases, but you
can tweak them to get different behavior.
+ */
+public class RackAwareWeightedLoadBalancingPolicy extends
DefaultLoadBalancingPolicy {
Review Comment:
Yes, it's certainly more valuable for cases where the rack->rack latency is
higher. However, being rack-aware is just one part of it, I think the mechanism
around preferring nodes with fewer in-flight requests is an improvement over
the round-robin mechanism (assuming non-token-aware, or all replicas) in
DefaultLoadBalancingPolicy/BasicLoadBalancingPolicy. Additionally, the
getAndUpdate in those balancers ends up being a bottleneck in high-throughput
situations. It's been a while since our ApacheCon talk and there's no recording
but the slides are available here
https://www.apachecon.com/acna2022/slides/04-Khaku-Lynch_Improving-Cassandra-Client.pdf
and they go into the problems a bit.
My eventual goal is to make this the default load balancing policy in the
driver but for now I'll settle for getting it in and making it an option.
Regarding the metrics - that's perhaps a little tricky since we're creating
a query plan here rather than firing off requests. Additionally, the
scoring/ordering is decoupled from the characteristics that resulted in the
score - perhaps some trace logging with the characteristics? In practice we saw
latencies drop immediately when we deployed this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
Re: [PR] Rack-aware weighted load balancing policy [cassandra-java-driver]
aratno commented on code in PR #1922:
URL:
https://github.com/apache/cassandra-java-driver/pull/1922#discussion_r1538077033
##
core/src/main/java/com/datastax/oss/driver/internal/core/loadbalancing/RackAwareWeightedLoadBalancingPolicy.java:
##
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.datastax.oss.driver.internal.core.loadbalancing;
+
+import static java.util.concurrent.TimeUnit.SECONDS;
+
+import com.datastax.oss.driver.api.core.config.DefaultDriverOption;
+import com.datastax.oss.driver.api.core.context.DriverContext;
+import com.datastax.oss.driver.api.core.metadata.Node;
+import com.datastax.oss.driver.api.core.session.Request;
+import com.datastax.oss.driver.api.core.session.Session;
+import com.datastax.oss.driver.internal.core.util.ArrayUtils;
+import com.datastax.oss.driver.internal.core.util.collection.QueryPlan;
+import com.datastax.oss.driver.internal.core.util.collection.SimpleQueryPlan;
+import edu.umd.cs.findbugs.annotations.NonNull;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Objects;
+import java.util.Queue;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A load balancing policy that optimally balances between sending load to
local token holder,
+ * rack replicas, and local datacenter replicas (in that order).
+ *
+ * The default weights are good for the vast majority of use cases, but you
can tweak them to get different behavior.
+ */
+public class RackAwareWeightedLoadBalancingPolicy extends
DefaultLoadBalancingPolicy {
Review Comment:
It looks like this is going to be most useful in environments like AWS,
where Cassandra racks correspond to AWS AZs, and there are relatively few
racks, and a high likelihood that an application has some instances running in
the same rack as Cassandra. In other environments where racks correspond to
actual racks in a datacenter, there are many more racks, and the likelihood of
an application running in the same rack is low, there's not as much benefit to
prioritizing rack-alignment as heavily.
Does that sound right based on your experience?
It would be useful to have a metric for the count of requests that go to
each category (rack-aligned, replica-aligned, instance-starting,
instance-unhealthy), so users can know whether they're setting local rack
correctly, set weights correctly, etc.
##
core/src/main/java/com/datastax/oss/driver/internal/core/loadbalancing/RackAwareWeightedLoadBalancingPolicy.java:
##
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.datastax.oss.driver.internal.core.loadbalancing;
+
+import static java.util.concurrent.TimeUnit.SECONDS;
+
+import com.datastax.oss.driver.api.core.config.DefaultDriverOption;
+import com.datastax.oss.driver.api.core.context.DriverContext;
+import com.datastax.oss.driver.api.core.metadata.Node;
+import com.datastax.oss.driver.api.core.session.Request;
+import com.datastax.oss.driver.api.core.session.Session;
+import com.datastax.oss.driver.internal.core.util.ArrayUtils;
+import com.datastax.oss.driver.internal.core.util.collection.QueryPlan;
+import com.datastax.oss.driver.internal.core.util.collection.SimpleQueryPlan;
+import edu.umd.cs.findbugs.annotations.NonNull;
+import java.util.ArrayList;
+import ja
[jira] [Commented] (CASSANDRA-19042) Repair fuzz tests fail with paxos_variant: v2
[
https://issues.apache.org/jira/browse/CASSANDRA-19042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830670#comment-17830670
]
Ekaterina Dimitrova commented on CASSANDRA-19042:
-
Overall LGTM (though limited knowledge in that part of the codebase, I admit).
Some nits left, which can be addressed on commit.
The CI run was limited preliminary development testing (the skinny dev workflow
Berenguer created).
We will need a full pre-commit CI. Also, the patch needs to be propagated to
trunk and tested there, too.
Do we want to run all tests with paxos_v2 enabled, at least on one of the
branches? WDYT?
Thanks
> Repair fuzz tests fail with paxos_variant: v2
> -
>
> Key: CASSANDRA-19042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19042
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Repair, Feature/Lightweight Transactions,
> Test/fuzz
>Reporter: Branimir Lambov
>Assignee: David Capwell
>Priority: Normal
> Fix For: 5.0-rc, 5.x
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Adding {{paxos_variant: v2}} to the test yaml causes all fuzz repair tests to
> fail with
> {code}
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.gms.EndpointStateSerializer.serializedSize(EndpointState.java:337)
> at
> org.apache.cassandra.gms.EndpointStateSerializer.serializedSize(EndpointState.java:300)
> at
> org.apache.cassandra.service.paxos.cleanup.PaxosStartPrepareCleanup$RequestSerializer.serializedSize(PaxosStartPrepareCleanup.java:176)
> at
> org.apache.cassandra.service.paxos.cleanup.PaxosStartPrepareCleanup$RequestSerializer.serializedSize(PaxosStartPrepareCleanup.java:147)
> at
> org.apache.cassandra.net.Message$Serializer.payloadSize(Message.java:1067)
> at org.apache.cassandra.net.Message.payloadSize(Message.java:1114)
> at
> org.apache.cassandra.net.Message$Serializer.serializedSize(Message.java:750)
> at org.apache.cassandra.net.Message.serializedSize(Message.java:1094)
> ...
> {code}
> This happens for all three options of {{paxos_state_purging}} and both with
> and without {{storage_compatibility_mode: NONE}}.
> Tests still fail if {{PaxosStartPrepareCleanup}} is changed to use
> {{EndpointState.nullableSerializer}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19332) Dropwizard Meter causes timeouts when infrequently used
[
https://issues.apache.org/jira/browse/CASSANDRA-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830659#comment-17830659
]
Maxim Muzafarov commented on CASSANDRA-19332:
-
My +1 here as well.
> Dropwizard Meter causes timeouts when infrequently used
> ---
>
> Key: CASSANDRA-19332
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19332
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0, 5.1
>
> Attachments: ci_summary_4.1.html, ci_summary_5.0.html,
> ci_summary_trunk.html, result_details_4.1.tar.gz, result_details_5.0.tar.gz,
> result_details_trunk.tar.gz
>
>
> Observed instances of timeouts on clusters with long uptime and infrequently
> used tables and possibly just request paths such as not using CAS for large
> fractions of a year.
> CAS seems to be more severely impacted because it has more metrics in the
> request path such as latency measurements for prepare, propose, and the read
> from the underlying table.
> Tracing showed ~600-800 milliseconds for these operations in between the
> “appending to memtable” and “sending a response” events. Reads had a delay
> between finishing the construction of the iterator and sending the read
> response.
> Stack traces dumped every 100 milliseconds using {{sjk}} shows that in
> prepare and propose a lot of time was being spent in
> {{{}Meter.tickIfNecessary{}}}.
> {code:java}
> Thread [2537] RUNNABLE at 2024-01-25T21:14:48.218 - MutationStage-2
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:71)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.prepare(PaxosState.java:92)
> Thread [2539] RUNNABLE at 2024-01-25T21:14:48.520 - MutationStage-4
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:72)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.propose(PaxosState.java:127){code}
> {{tickIfNecessary}} does a linear amount of work proportional to the time
> since the last time the metric was updated/read/created and this can actually
> take a measurable amount of time even in a tight loop. On my M2 MBP it was
> 1.5 milliseconds for a day, ~200 days took ~74 milliseconds. Before it warmed
> up it was 140 milliseconds.
> A quick fix is to schedule a task to read all the meters once a day so it
> isn’t done in the request path and we have a more incremental amount to
> process at a time.
> Also observed that {{tickIfNecessary}} is not 100% thread safe in that if it
> takes longer than 5 seconds to run the loop it can end up with multiple
> threads attempting to run the loop at once and then they will concurrently
> run {{EWMA.tick}} which probably results in some ticks not being performed.
> This issue is still present in the latest version of {{Metrics}} if using
> {{{}EWMA{}}}, but {{SlidingWindowTimeAverages}} looks like it has a bounded
> amount of work required to tick. Switching would change how our metrics work
> since the two don't have the same behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19332) Dropwizard Meter causes timeouts when infrequently used
[
https://issues.apache.org/jira/browse/CASSANDRA-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830654#comment-17830654
]
Stefan Miklosovic commented on CASSANDRA-19332:
---
[~mmuzaf] would you please explicitly +1 this, I just stumbled upon this ...
you have indeed better visibility into this. Thank you very much.
> Dropwizard Meter causes timeouts when infrequently used
> ---
>
> Key: CASSANDRA-19332
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19332
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0, 5.1
>
> Attachments: ci_summary_4.1.html, ci_summary_5.0.html,
> ci_summary_trunk.html, result_details_4.1.tar.gz, result_details_5.0.tar.gz,
> result_details_trunk.tar.gz
>
>
> Observed instances of timeouts on clusters with long uptime and infrequently
> used tables and possibly just request paths such as not using CAS for large
> fractions of a year.
> CAS seems to be more severely impacted because it has more metrics in the
> request path such as latency measurements for prepare, propose, and the read
> from the underlying table.
> Tracing showed ~600-800 milliseconds for these operations in between the
> “appending to memtable” and “sending a response” events. Reads had a delay
> between finishing the construction of the iterator and sending the read
> response.
> Stack traces dumped every 100 milliseconds using {{sjk}} shows that in
> prepare and propose a lot of time was being spent in
> {{{}Meter.tickIfNecessary{}}}.
> {code:java}
> Thread [2537] RUNNABLE at 2024-01-25T21:14:48.218 - MutationStage-2
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:71)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.prepare(PaxosState.java:92)
> Thread [2539] RUNNABLE at 2024-01-25T21:14:48.520 - MutationStage-4
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:72)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.propose(PaxosState.java:127){code}
> {{tickIfNecessary}} does a linear amount of work proportional to the time
> since the last time the metric was updated/read/created and this can actually
> take a measurable amount of time even in a tight loop. On my M2 MBP it was
> 1.5 milliseconds for a day, ~200 days took ~74 milliseconds. Before it warmed
> up it was 140 milliseconds.
> A quick fix is to schedule a task to read all the meters once a day so it
> isn’t done in the request path and we have a more incremental amount to
> process at a time.
> Also observed that {{tickIfNecessary}} is not 100% thread safe in that if it
> takes longer than 5 seconds to run the loop it can end up with multiple
> threads attempting to run the loop at once and then they will concurrently
> run {{EWMA.tick}} which probably results in some ticks not being performed.
> This issue is still present in the latest version of {{Metrics}} if using
> {{{}EWMA{}}}, but {{SlidingWindowTimeAverages}} looks like it has a bounded
> amount of work required to tick. Switching would change how our metrics work
> since the two don't have the same behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19332) Dropwizard Meter causes timeouts when infrequently used
[
https://issues.apache.org/jira/browse/CASSANDRA-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830652#comment-17830652
]
Ariel Weisberg commented on CASSANDRA-19332:
I think I figured out what happened. The latest test repo version was tested,
but my branch was behind trunk so it didn't have the change to nodetool.
> Dropwizard Meter causes timeouts when infrequently used
> ---
>
> Key: CASSANDRA-19332
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19332
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0, 5.1
>
> Attachments: ci_summary_4.1.html, ci_summary_5.0.html,
> ci_summary_trunk.html, result_details_4.1.tar.gz, result_details_5.0.tar.gz,
> result_details_trunk.tar.gz
>
>
> Observed instances of timeouts on clusters with long uptime and infrequently
> used tables and possibly just request paths such as not using CAS for large
> fractions of a year.
> CAS seems to be more severely impacted because it has more metrics in the
> request path such as latency measurements for prepare, propose, and the read
> from the underlying table.
> Tracing showed ~600-800 milliseconds for these operations in between the
> “appending to memtable” and “sending a response” events. Reads had a delay
> between finishing the construction of the iterator and sending the read
> response.
> Stack traces dumped every 100 milliseconds using {{sjk}} shows that in
> prepare and propose a lot of time was being spent in
> {{{}Meter.tickIfNecessary{}}}.
> {code:java}
> Thread [2537] RUNNABLE at 2024-01-25T21:14:48.218 - MutationStage-2
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:71)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.prepare(PaxosState.java:92)
> Thread [2539] RUNNABLE at 2024-01-25T21:14:48.520 - MutationStage-4
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:72)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.propose(PaxosState.java:127){code}
> {{tickIfNecessary}} does a linear amount of work proportional to the time
> since the last time the metric was updated/read/created and this can actually
> take a measurable amount of time even in a tight loop. On my M2 MBP it was
> 1.5 milliseconds for a day, ~200 days took ~74 milliseconds. Before it warmed
> up it was 140 milliseconds.
> A quick fix is to schedule a task to read all the meters once a day so it
> isn’t done in the request path and we have a more incremental amount to
> process at a time.
> Also observed that {{tickIfNecessary}} is not 100% thread safe in that if it
> takes longer than 5 seconds to run the loop it can end up with multiple
> threads attempting to run the loop at once and then they will concurrently
> run {{EWMA.tick}} which probably results in some ticks not being performed.
> This issue is still present in the latest version of {{Metrics}} if using
> {{{}EWMA{}}}, but {{SlidingWindowTimeAverages}} looks like it has a bounded
> amount of work required to tick. Switching would change how our metrics work
> since the two don't have the same behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19332) Dropwizard Meter causes timeouts when infrequently used
[
https://issues.apache.org/jira/browse/CASSANDRA-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830649#comment-17830649
]
Ariel Weisberg commented on CASSANDRA-19332:
So which build are you referring to? When I look at the trunk build that
generated that error the log has
Cloned https://github.com/apache/cassandra-dtest.git trunk to
/workspace/context/cassandra-dtest; commit is
7a82b3757c136f79b52a76fdf3e98891dfff6b41
and that SHA comes after f3ca59c
[https://github.com/apache/cassandra-dtest/commits/trunk/]
I'll look a little deeper and try to figure out how the executors ended up
running with the wrong commit on the dtest repo because clearly it happened.
> Dropwizard Meter causes timeouts when infrequently used
> ---
>
> Key: CASSANDRA-19332
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19332
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0, 5.1
>
> Attachments: ci_summary_4.1.html, ci_summary_5.0.html,
> ci_summary_trunk.html, result_details_4.1.tar.gz, result_details_5.0.tar.gz,
> result_details_trunk.tar.gz
>
>
> Observed instances of timeouts on clusters with long uptime and infrequently
> used tables and possibly just request paths such as not using CAS for large
> fractions of a year.
> CAS seems to be more severely impacted because it has more metrics in the
> request path such as latency measurements for prepare, propose, and the read
> from the underlying table.
> Tracing showed ~600-800 milliseconds for these operations in between the
> “appending to memtable” and “sending a response” events. Reads had a delay
> between finishing the construction of the iterator and sending the read
> response.
> Stack traces dumped every 100 milliseconds using {{sjk}} shows that in
> prepare and propose a lot of time was being spent in
> {{{}Meter.tickIfNecessary{}}}.
> {code:java}
> Thread [2537] RUNNABLE at 2024-01-25T21:14:48.218 - MutationStage-2
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:71)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.prepare(PaxosState.java:92)
> Thread [2539] RUNNABLE at 2024-01-25T21:14:48.520 - MutationStage-4
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:72)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.propose(PaxosState.java:127){code}
> {{tickIfNecessary}} does a linear amount of work proportional to the time
> since the last time the metric was updated/read/created and this can actually
> take a measurable amount of time even in a tight loop. On my M2 MBP it was
> 1.5 milliseconds for a day, ~200 days took ~74 milliseconds. Before it warmed
> up it was 140 milliseconds.
> A quick fix is to schedule a task to read all the meters once a day so it
> isn’t done in the request path and we have a more incremental amount to
> process at a time.
> Also observed that {{tickIfNecessary}} is not 100% thread safe in that if it
> takes longer than 5 seconds to run the loop it can end up with multiple
> threads attempting to run the loop at once and then they will concurrently
> run {{EWMA.tick}} which probably results in some ticks not being performed.
> This issue is still present in the latest version of {{Metrics}} if using
> {{{}EWMA{}}}, but {{SlidingWindowTimeAverages}} looks like it has a bounded
> amount of work required to tick. Switching would change how our metrics work
> since the two don't have the same behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
Re: [PR] Rack-aware weighted load balancing policy [cassandra-java-driver]
aratno commented on PR #1922: URL: https://github.com/apache/cassandra-java-driver/pull/1922#issuecomment-2018636818 > The documentation could probably use some more work, but I'd love guidance on where to focus that: the implementation class? Or reference.conf? I'd put configuration documentation in `reference.conf`. For more context on why someone would benefit from this LBP over the default, put that in `manual/core/load_balancing/README.md`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
Re: [PR] Rack-aware weighted load balancing policy [cassandra-java-driver]
akhaku commented on PR #1922: URL: https://github.com/apache/cassandra-java-driver/pull/1922#issuecomment-2018549946 Additionally, this will tackle JAVA-3040 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[PR] Rack-aware weighted load balancing policy [cassandra-java-driver]
akhaku opened a new pull request, #1922: URL: https://github.com/apache/cassandra-java-driver/pull/1922 This PR open-sources the work Joey Lynch and I did and presented at ApacheCon 2022. It requires a bit of work before it's mergeable, but hoping to get some feedback on a couple areas before I put in much more work. A couple major things left to do: - Port the tests we added internally. We also have some jmh tests but I'm not sure if those will fit in the driver codebase - Pass the local rack through to the driver. Internally we passed it through the context, for OSS perhaps the right thing it to implement something like the optional/required local DC pattern. - The documentation could probably use some more work, but I'd love guidance on where to focus that: the implementation class? Or reference.conf? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-19332) Dropwizard Meter causes timeouts when infrequently used
[
https://issues.apache.org/jira/browse/CASSANDRA-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830611#comment-17830611
]
Stefan Miklosovic edited comment on CASSANDRA-19332 at 3/25/24 5:23 PM:
j17 precommit tests (minus upgrade)
[https://app.circleci.com/pipelines/github/instaclustr/cassandra/4080/workflows/677745ba-4520-4e1d-951f-1829004d3235]
btw I just realized that I will not be able to run dtests because my CircleCI
plan is timeouting all jobs after 1h and I do not have parallelism big enough
to fit it into that limit either.
from your test results for trunk I read:
{code:java}
ccmlib.node.ToolError: Subprocess ['nodetool', '-h', 'localhost', '-p', '7100',
'cms', 'initialize'] exited with non-zero status; exit status: 1; stdout:
nodetool: Found unexpected parameters: [cms, initialize] See 'nodetool help' or
'nodetool help '. ; stderr: Picked up JAVA_TOOL_OPTIONS:
-XX:ErrorFile=/parallel-ci/output/hs_err_pid_%p.log
{code}
this is caused by you running old dtests, you probably havent pulled
cassandra-dtest repo
[https://github.com/apache/cassandra-dtest/commit/f3ca59c76dc946e75b3b9dbc90f22dc51bf1c73a]
as i read it from the logs you uploaded, all dtest failures for trunk are about
this
was (Author: smiklosovic):
j17 precommit tests (minus upgrade)
https://app.circleci.com/pipelines/github/instaclustr/cassandra/4080/workflows/677745ba-4520-4e1d-951f-1829004d3235
from your test results for trunk I read:
{code}
ccmlib.node.ToolError: Subprocess ['nodetool', '-h', 'localhost', '-p', '7100',
'cms', 'initialize'] exited with non-zero status; exit status: 1; stdout:
nodetool: Found unexpected parameters: [cms, initialize] See 'nodetool help' or
'nodetool help '. ; stderr: Picked up JAVA_TOOL_OPTIONS:
-XX:ErrorFile=/parallel-ci/output/hs_err_pid_%p.log
{code}
this is caused by you running old dtests, you probably havent pulled
cassandra-dtest repo
https://github.com/apache/cassandra-dtest/commit/f3ca59c76dc946e75b3b9dbc90f22dc51bf1c73a
as i read it from the logs you uploaded, all dtest failures for trunk are
about this
> Dropwizard Meter causes timeouts when infrequently used
> ---
>
> Key: CASSANDRA-19332
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19332
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0, 5.1
>
> Attachments: ci_summary_4.1.html, ci_summary_5.0.html,
> ci_summary_trunk.html, result_details_4.1.tar.gz, result_details_5.0.tar.gz,
> result_details_trunk.tar.gz
>
>
> Observed instances of timeouts on clusters with long uptime and infrequently
> used tables and possibly just request paths such as not using CAS for large
> fractions of a year.
> CAS seems to be more severely impacted because it has more metrics in the
> request path such as latency measurements for prepare, propose, and the read
> from the underlying table.
> Tracing showed ~600-800 milliseconds for these operations in between the
> “appending to memtable” and “sending a response” events. Reads had a delay
> between finishing the construction of the iterator and sending the read
> response.
> Stack traces dumped every 100 milliseconds using {{sjk}} shows that in
> prepare and propose a lot of time was being spent in
> {{{}Meter.tickIfNecessary{}}}.
> {code:java}
> Thread [2537] RUNNABLE at 2024-01-25T21:14:48.218 - MutationStage-2
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:71)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.prepare(PaxosState.java:92)
> Thread [2539] RUNNABLE at 2024-01-25T21:14:48.520 - MutationStage-4
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:72)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.propose(PaxosState.java:127){code}
> {{tickIfNecessary}} does a linear amount of work proportional to the time
> since the last time the metric was updated/read/created and this can actually
> take a measurable amount of time even in a tight loop. On my M2 MBP it was
> 1.5 milliseconds for a day, ~200 days took ~74 milliseconds. Before it warmed
> up it was 140 milliseconds
[jira] [Commented] (CASSANDRA-19332) Dropwizard Meter causes timeouts when infrequently used
[
https://issues.apache.org/jira/browse/CASSANDRA-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830611#comment-17830611
]
Stefan Miklosovic commented on CASSANDRA-19332:
---
j17 precommit tests (minus upgrade)
https://app.circleci.com/pipelines/github/instaclustr/cassandra/4080/workflows/677745ba-4520-4e1d-951f-1829004d3235
from your test results for trunk I read:
{code}
ccmlib.node.ToolError: Subprocess ['nodetool', '-h', 'localhost', '-p', '7100',
'cms', 'initialize'] exited with non-zero status; exit status: 1; stdout:
nodetool: Found unexpected parameters: [cms, initialize] See 'nodetool help' or
'nodetool help '. ; stderr: Picked up JAVA_TOOL_OPTIONS:
-XX:ErrorFile=/parallel-ci/output/hs_err_pid_%p.log
{code}
this is caused by you running old dtests, you probably havent pulled
cassandra-dtest repo
https://github.com/apache/cassandra-dtest/commit/f3ca59c76dc946e75b3b9dbc90f22dc51bf1c73a
as i read it from the logs you uploaded, all dtest failures for trunk are
about this
> Dropwizard Meter causes timeouts when infrequently used
> ---
>
> Key: CASSANDRA-19332
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19332
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0, 5.1
>
> Attachments: ci_summary_4.1.html, ci_summary_5.0.html,
> ci_summary_trunk.html, result_details_4.1.tar.gz, result_details_5.0.tar.gz,
> result_details_trunk.tar.gz
>
>
> Observed instances of timeouts on clusters with long uptime and infrequently
> used tables and possibly just request paths such as not using CAS for large
> fractions of a year.
> CAS seems to be more severely impacted because it has more metrics in the
> request path such as latency measurements for prepare, propose, and the read
> from the underlying table.
> Tracing showed ~600-800 milliseconds for these operations in between the
> “appending to memtable” and “sending a response” events. Reads had a delay
> between finishing the construction of the iterator and sending the read
> response.
> Stack traces dumped every 100 milliseconds using {{sjk}} shows that in
> prepare and propose a lot of time was being spent in
> {{{}Meter.tickIfNecessary{}}}.
> {code:java}
> Thread [2537] RUNNABLE at 2024-01-25T21:14:48.218 - MutationStage-2
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:71)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.prepare(PaxosState.java:92)
> Thread [2539] RUNNABLE at 2024-01-25T21:14:48.520 - MutationStage-4
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:72)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.propose(PaxosState.java:127){code}
> {{tickIfNecessary}} does a linear amount of work proportional to the time
> since the last time the metric was updated/read/created and this can actually
> take a measurable amount of time even in a tight loop. On my M2 MBP it was
> 1.5 milliseconds for a day, ~200 days took ~74 milliseconds. Before it warmed
> up it was 140 milliseconds.
> A quick fix is to schedule a task to read all the meters once a day so it
> isn’t done in the request path and we have a more incremental amount to
> process at a time.
> Also observed that {{tickIfNecessary}} is not 100% thread safe in that if it
> takes longer than 5 seconds to run the loop it can end up with multiple
> threads attempting to run the loop at once and then they will concurrently
> run {{EWMA.tick}} which probably results in some ticks not being performed.
> This issue is still present in the latest version of {{Metrics}} if using
> {{{}EWMA{}}}, but {{SlidingWindowTimeAverages}} looks like it has a bounded
> amount of work required to tick. Switching would change how our metrics work
> since the two don't have the same behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRASC-21) Include sidecar clients with build artifacts
[ https://issues.apache.org/jira/browse/CASSANDRASC-21?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Haddad reassigned CASSANDRASC-21: - Assignee: (was: Jon Haddad) > Include sidecar clients with build artifacts > - > > Key: CASSANDRASC-21 > URL: https://issues.apache.org/jira/browse/CASSANDRASC-21 > Project: Sidecar for Apache Cassandra > Issue Type: Improvement >Reporter: Jon Haddad >Priority: Normal > > Swagger can generate client libraries through gradle. We should bundle the > clients with the build artifacts so people don't have to download them > separately. > We can leverage this for CASSANDRASC-19 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-12881) Move SASI docs into sphinx docs
[ https://issues.apache.org/jira/browse/CASSANDRA-12881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Haddad reassigned CASSANDRA-12881: -- Assignee: (was: Jon Haddad) > Move SASI docs into sphinx docs > --- > > Key: CASSANDRA-12881 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12881 > Project: Cassandra > Issue Type: Task > Components: Legacy/Documentation and Website >Reporter: Evan Prothro >Priority: Low > Labels: lhf, sasi > > Previous TODO in code regarding SASI docs: > TODO: we should probably move the first half of that documentation to the > general documentation, and the implementation explanation parts into the wiki. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-14342) Refactor getColumnFamilyStore() to getTable()
[ https://issues.apache.org/jira/browse/CASSANDRA-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Haddad reassigned CASSANDRA-14342: -- Assignee: (was: Jon Haddad) > Refactor getColumnFamilyStore() to getTable() > - > > Key: CASSANDRA-14342 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14342 > Project: Cassandra > Issue Type: Sub-task > Components: Legacy/Local Write-Read Paths >Reporter: Jon Haddad >Priority: Normal > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-14340) Refactor ColumnFamilyStore to Table
[ https://issues.apache.org/jira/browse/CASSANDRA-14340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Haddad reassigned CASSANDRA-14340: -- Assignee: (was: Jon Haddad) > Refactor ColumnFamilyStore to Table > --- > > Key: CASSANDRA-14340 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14340 > Project: Cassandra > Issue Type: Sub-task > Components: Legacy/Core >Reporter: Jon Haddad >Priority: Normal > > This end result of this should change the ColumnFamily store, > ColumnFamilyStoreMBean, and tests that reference them by name. > Deserves a note in news as this will break JMX compatibility and affect > scripts which change log level by class. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-14339) Modernize Internal Vocabulary
[ https://issues.apache.org/jira/browse/CASSANDRA-14339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Haddad reassigned CASSANDRA-14339: -- Assignee: (was: Jon Haddad) > Modernize Internal Vocabulary > - > > Key: CASSANDRA-14339 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14339 > Project: Cassandra > Issue Type: Task > Components: Legacy/Core >Reporter: Jon Haddad >Priority: Normal > Fix For: 5.x > > > This is an umbrella ticket for all modifications necessary to update the code > base from old Thrift-centric terminology to CQL terminology. References to > Column families should be completely removed at the close of this issue, rows > should mean CQL rows, and "wide row" = partition. > Each sub ticket should be as small of a refactor as possible. For instance, > the rename of a single class should only touch the class, with a follow up > ticket for the variable names, methods, etc, to keep the scope of each > refactor to a minimum. The exception to this are the comments that are > associated with a block of code, which should be addressed with the ticket. > Ensure changes that affect JMX are thoroughly tested, this should be covered > by our utests & dtests, but care should be taken in case they are not. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15080) Paxos tables should allow a configurable chunk length
[ https://issues.apache.org/jira/browse/CASSANDRA-15080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Haddad reassigned CASSANDRA-15080: -- Assignee: (was: Jon Haddad) > Paxos tables should allow a configurable chunk length > - > > Key: CASSANDRA-15080 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15080 > Project: Cassandra > Issue Type: Improvement > Components: Feature/Lightweight Transactions >Reporter: Jon Haddad >Priority: Normal > Labels: performance > Attachments: flamegraph-bad-perf.svg > > > In doing some research for a client on LWTs, I found that once we start > pushing a node hard enough, with enough LWTs, decompressing the data in the > paxos table takes up quite a bit of time. I've attached an SVG flame graph > showing about 10% of time is spent in LZ4_decompress_fast in queries hitting > the paxos table. > We should allow the user to modify the compression settings on this table. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19042) Repair fuzz tests fail with paxos_variant: v2
[
https://issues.apache.org/jira/browse/CASSANDRA-19042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Blake Eggleston updated CASSANDRA-19042:
Status: Ready to Commit (was: Review In Progress)
LGTM, thanks!
> Repair fuzz tests fail with paxos_variant: v2
> -
>
> Key: CASSANDRA-19042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19042
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Repair, Feature/Lightweight Transactions,
> Test/fuzz
>Reporter: Branimir Lambov
>Assignee: David Capwell
>Priority: Normal
> Fix For: 5.0-rc, 5.x
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Adding {{paxos_variant: v2}} to the test yaml causes all fuzz repair tests to
> fail with
> {code}
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.gms.EndpointStateSerializer.serializedSize(EndpointState.java:337)
> at
> org.apache.cassandra.gms.EndpointStateSerializer.serializedSize(EndpointState.java:300)
> at
> org.apache.cassandra.service.paxos.cleanup.PaxosStartPrepareCleanup$RequestSerializer.serializedSize(PaxosStartPrepareCleanup.java:176)
> at
> org.apache.cassandra.service.paxos.cleanup.PaxosStartPrepareCleanup$RequestSerializer.serializedSize(PaxosStartPrepareCleanup.java:147)
> at
> org.apache.cassandra.net.Message$Serializer.payloadSize(Message.java:1067)
> at org.apache.cassandra.net.Message.payloadSize(Message.java:1114)
> at
> org.apache.cassandra.net.Message$Serializer.serializedSize(Message.java:750)
> at org.apache.cassandra.net.Message.serializedSize(Message.java:1094)
> ...
> {code}
> This happens for all three options of {{paxos_state_purging}} and both with
> and without {{storage_compatibility_mode: NONE}}.
> Tests still fail if {{PaxosStartPrepareCleanup}} is changed to use
> {{EndpointState.nullableSerializer}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14095) New configuration settings using duration types
[
https://issues.apache.org/jira/browse/CASSANDRA-14095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jon Haddad updated CASSANDRA-14095:
---
Resolution: Duplicate
Status: Resolved (was: Open)
> New configuration settings using duration types
> ---
>
> Key: CASSANDRA-14095
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14095
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Jon Haddad
>Assignee: Jon Haddad
>Priority: Normal
>
> All the _ms configuration params are difficult to work with. We can allow
> users to provide a duration type, like 3h, for settings instead of requiring
> _ms based settings.
> This first jira is to allow for cassandra.yaml to take duration types instead
> of ms based settings.
> Given a setting, blah_ms, the new setting should be {{blah}} and require a
> duration like {{3h}}. The old _ms settings should continue to work and error
> if both are used.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-14343) CF -> Tablename Comment cleanup
[ https://issues.apache.org/jira/browse/CASSANDRA-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Haddad reassigned CASSANDRA-14343: -- Assignee: (was: Jon Haddad) > CF -> Tablename Comment cleanup > --- > > Key: CASSANDRA-14343 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14343 > Project: Cassandra > Issue Type: Sub-task > Components: Legacy/Core >Reporter: Jon Haddad >Priority: Normal > > There's going to be sporadic comments in the codebase referring to column > families, these should be updated to tables. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-13890) Expose current compaction throughput
[
https://issues.apache.org/jira/browse/CASSANDRA-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jon Haddad reassigned CASSANDRA-13890:
--
Assignee: (was: Jon Haddad)
> Expose current compaction throughput
>
>
> Key: CASSANDRA-13890
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13890
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Tools
>Reporter: Jon Haddad
>Priority: Low
> Labels: lhf
> Fix For: 5.x
>
>
> Getting and setting the current compaction throughput limit is supported, but
> there's no means of knowing if the setting is actually making a difference.
> Let's expose the current throughput being utilized by Cassandra that's in the
> {{compactionRateLimiter}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-13820) List languages on driver list alphabetically
[ https://issues.apache.org/jira/browse/CASSANDRA-13820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Haddad reassigned CASSANDRA-13820: -- Assignee: (was: Jon Haddad) > List languages on driver list alphabetically > - > > Key: CASSANDRA-13820 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13820 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Documentation and Website >Reporter: Jon Haddad >Priority: Low > > This is pretty minor, but I think the list of drivers on > https://cassandra.apache.org/doc/latest/getting_started/index.html should be > listed in alphabetical order to make it more readable. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15144) documentation lists old version of bcc tools
[
https://issues.apache.org/jira/browse/CASSANDRA-15144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jon Haddad reassigned CASSANDRA-15144:
--
Assignee: (was: Jon Haddad)
> documentation lists old version of bcc tools
>
>
> Key: CASSANDRA-15144
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15144
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Jon Haddad
>Priority: Normal
>
> In our
> [docs|http://cassandra.apache.org/doc/latest/troubleshooting/use_tools.html]
> we list this:
> {code}
> apt install bcc-tools
> {code}
> In Ubuntu 18 this was moved, and we should expand the instructions with the
> following:
> {code}
> apt install bpfcc-tools
> {code}
> Additionally, the names of the tools have changed:
> {code}
> $ cachestat-bpfcc -T 1
> TIME HITS MISSES DIRTIES READ_HIT% WRITE_HIT% BUFFERS_MB
> CACHED_MB
> 16:03:33 5279 6115 5271 0.1% 28.7% 41
> 7590
> 16:03:34344292281526237 14.3% 19.9% 41
> 7635
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19332) Dropwizard Meter causes timeouts when infrequently used
[
https://issues.apache.org/jira/browse/CASSANDRA-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830606#comment-17830606
]
Ariel Weisberg commented on CASSANDRA-19332:
Sounds good, I will wait for that.
> Dropwizard Meter causes timeouts when infrequently used
> ---
>
> Key: CASSANDRA-19332
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19332
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0, 5.1
>
> Attachments: ci_summary_4.1.html, ci_summary_5.0.html,
> ci_summary_trunk.html, result_details_4.1.tar.gz, result_details_5.0.tar.gz,
> result_details_trunk.tar.gz
>
>
> Observed instances of timeouts on clusters with long uptime and infrequently
> used tables and possibly just request paths such as not using CAS for large
> fractions of a year.
> CAS seems to be more severely impacted because it has more metrics in the
> request path such as latency measurements for prepare, propose, and the read
> from the underlying table.
> Tracing showed ~600-800 milliseconds for these operations in between the
> “appending to memtable” and “sending a response” events. Reads had a delay
> between finishing the construction of the iterator and sending the read
> response.
> Stack traces dumped every 100 milliseconds using {{sjk}} shows that in
> prepare and propose a lot of time was being spent in
> {{{}Meter.tickIfNecessary{}}}.
> {code:java}
> Thread [2537] RUNNABLE at 2024-01-25T21:14:48.218 - MutationStage-2
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:71)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.prepare(PaxosState.java:92)
> Thread [2539] RUNNABLE at 2024-01-25T21:14:48.520 - MutationStage-4
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:72)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.propose(PaxosState.java:127){code}
> {{tickIfNecessary}} does a linear amount of work proportional to the time
> since the last time the metric was updated/read/created and this can actually
> take a measurable amount of time even in a tight loop. On my M2 MBP it was
> 1.5 milliseconds for a day, ~200 days took ~74 milliseconds. Before it warmed
> up it was 140 milliseconds.
> A quick fix is to schedule a task to read all the meters once a day so it
> isn’t done in the request path and we have a more incremental amount to
> process at a time.
> Also observed that {{tickIfNecessary}} is not 100% thread safe in that if it
> takes longer than 5 seconds to run the loop it can end up with multiple
> threads attempting to run the loop at once and then they will concurrently
> run {{EWMA.tick}} which probably results in some ticks not being performed.
> This issue is still present in the latest version of {{Metrics}} if using
> {{{}EWMA{}}}, but {{SlidingWindowTimeAverages}} looks like it has a bounded
> amount of work required to tick. Switching would change how our metrics work
> since the two don't have the same behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19332) Dropwizard Meter causes timeouts when infrequently used
[
https://issues.apache.org/jira/browse/CASSANDRA-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830602#comment-17830602
]
Stefan Miklosovic commented on CASSANDRA-19332:
---
yeah that makes sense, I can run a CI for trunk in Circle for you to see if
your CI was just a flaky or what ... give me some time here please.
> Dropwizard Meter causes timeouts when infrequently used
> ---
>
> Key: CASSANDRA-19332
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19332
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0, 5.1
>
> Attachments: ci_summary_4.1.html, ci_summary_5.0.html,
> ci_summary_trunk.html, result_details_4.1.tar.gz, result_details_5.0.tar.gz,
> result_details_trunk.tar.gz
>
>
> Observed instances of timeouts on clusters with long uptime and infrequently
> used tables and possibly just request paths such as not using CAS for large
> fractions of a year.
> CAS seems to be more severely impacted because it has more metrics in the
> request path such as latency measurements for prepare, propose, and the read
> from the underlying table.
> Tracing showed ~600-800 milliseconds for these operations in between the
> “appending to memtable” and “sending a response” events. Reads had a delay
> between finishing the construction of the iterator and sending the read
> response.
> Stack traces dumped every 100 milliseconds using {{sjk}} shows that in
> prepare and propose a lot of time was being spent in
> {{{}Meter.tickIfNecessary{}}}.
> {code:java}
> Thread [2537] RUNNABLE at 2024-01-25T21:14:48.218 - MutationStage-2
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:71)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.prepare(PaxosState.java:92)
> Thread [2539] RUNNABLE at 2024-01-25T21:14:48.520 - MutationStage-4
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:72)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.propose(PaxosState.java:127){code}
> {{tickIfNecessary}} does a linear amount of work proportional to the time
> since the last time the metric was updated/read/created and this can actually
> take a measurable amount of time even in a tight loop. On my M2 MBP it was
> 1.5 milliseconds for a day, ~200 days took ~74 milliseconds. Before it warmed
> up it was 140 milliseconds.
> A quick fix is to schedule a task to read all the meters once a day so it
> isn’t done in the request path and we have a more incremental amount to
> process at a time.
> Also observed that {{tickIfNecessary}} is not 100% thread safe in that if it
> takes longer than 5 seconds to run the loop it can end up with multiple
> threads attempting to run the loop at once and then they will concurrently
> run {{EWMA.tick}} which probably results in some ticks not being performed.
> This issue is still present in the latest version of {{Metrics}} if using
> {{{}EWMA{}}}, but {{SlidingWindowTimeAverages}} looks like it has a bounded
> amount of work required to tick. Switching would change how our metrics work
> since the two don't have the same behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13855) Implement Http Seed provider
[ https://issues.apache.org/jira/browse/CASSANDRA-13855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830597#comment-17830597 ] Maxim Muzafarov edited comment on CASSANDRA-13855 at 3/25/24 4:59 PM: -- I've left some comments on the PR. was (Author: mmuzaf): I've lest some comments to the PR. > Implement Http Seed provider > > > Key: CASSANDRA-13855 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13855 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Coordination, Legacy/Core >Reporter: Jon Haddad >Assignee: Claude Warren >Priority: Low > Labels: lhf > Fix For: 5.x > > Attachments: 0001-Add-URL-Seed-Provider-trunk.txt, signature.asc, > signature.asc, signature.asc > > Time Spent: 4h > Remaining Estimate: 0h > > Seems like including a dead simple seed provider that can fetch from a URL, 1 > line per seed, would be useful. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13855) Implement Http Seed provider
[ https://issues.apache.org/jira/browse/CASSANDRA-13855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830597#comment-17830597 ] Maxim Muzafarov commented on CASSANDRA-13855: - I've lest some comments to the PR. > Implement Http Seed provider > > > Key: CASSANDRA-13855 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13855 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Coordination, Legacy/Core >Reporter: Jon Haddad >Assignee: Claude Warren >Priority: Low > Labels: lhf > Fix For: 5.x > > Attachments: 0001-Add-URL-Seed-Provider-trunk.txt, signature.asc, > signature.asc, signature.asc > > Time Spent: 4h > Remaining Estimate: 0h > > Seems like including a dead simple seed provider that can fetch from a URL, 1 > line per seed, would be useful. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19429) Remove lock contention generated by getCapacity function in SSTableReader
[
https://issues.apache.org/jira/browse/CASSANDRA-19429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830582#comment-17830582
]
Dipietro Salvatore commented on CASSANDRA-19429:
[~rustyrazorblade] sure, let me know when you will be able to work on it so
that I can provide that
> Remove lock contention generated by getCapacity function in SSTableReader
> -
>
> Key: CASSANDRA-19429
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19429
> Project: Cassandra
> Issue Type: Bug
> Components: Local/SSTable
>Reporter: Dipietro Salvatore
>Assignee: Dipietro Salvatore
>Priority: Normal
> Fix For: 4.0.x, 4.1.x
>
> Attachments: Screenshot 2024-02-26 at 10.27.10.png, Screenshot
> 2024-02-27 at 11.29.41.png, Screenshot 2024-03-19 at 15.22.50.png,
> asprof_cass4.1.3__lock_20240216052912lock.html,
> image-2024-03-08-15-51-30-439.png, image-2024-03-08-15-52-07-902.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Profiling Cassandra 4.1.3 on large AWS instances, a high number of lock
> acquires is measured in the `getCapacity` function from
> `org/apache/cassandra/cache/InstrumentingCache` (1.9M lock acquires per 60
> seconds). Based on our tests on r8g.24xlarge instances (using Ubuntu 22.04),
> this limits the CPU utilization of the system to under 50% when testing at
> full load and therefore limits the achieved throughput.
> Removing the lock contention from the SSTableReader.java file by replacing
> the call to `getCapacity` with `size` achieves up to 2.95x increase in
> throughput on r8g.24xlarge and 2x on r7i.24xlarge:
> |Instance type|Cass 4.1.3|Cass 4.1.3 patched|
> |r8g.24xlarge|168k ops|496k ops (2.95x)|
> |r7i.24xlarge|153k ops|304k ops (1.98x)|
>
> Instructions to reproduce:
> {code:java}
> ## Requirements for Ubuntu 22.04
> sudo apt install -y ant git openjdk-11-jdk
> ## Build and run
> CASSANDRA_USE_JDK11=true ant realclean && CASSANDRA_USE_JDK11=true ant jar &&
> CASSANDRA_USE_JDK11=true ant stress-build && rm -rf data && bin/cassandra -f
> -R
> # Run
> bin/cqlsh -e 'drop table if exists keyspace1.standard1;' && \
> bin/cqlsh -e 'drop keyspace if exists keyspace1;' && \
> bin/nodetool clearsnapshot --all && tools/bin/cassandra-stress write
> n=1000 cl=ONE -rate threads=384 -node 127.0.0.1 -log file=cload.log
> -graph file=cload.html && \
> bin/nodetool compact keyspace1 && sleep 30s && \
> tools/bin/cassandra-stress mixed ratio\(write=10,read=90\) duration=10m
> cl=ONE -rate threads=406 -node localhost -log file=result.log -graph
> file=graph.html
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19332) Dropwizard Meter causes timeouts when infrequently used
[
https://issues.apache.org/jira/browse/CASSANDRA-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830574#comment-17830574
]
Ariel Weisberg commented on CASSANDRA-19332:
Yeah the Dropwizard fix makes this not necessary by fixing the cost of marking
a meter that hasn't been used in a long time. We can remove this once that is
fixed. Since we don't update dependencies in older versions so we will need
this in 4.0/4.1, and 5.0 if it releases before Dropwizard does.
> Dropwizard Meter causes timeouts when infrequently used
> ---
>
> Key: CASSANDRA-19332
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19332
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0, 5.1
>
> Attachments: ci_summary_4.1.html, ci_summary_5.0.html,
> ci_summary_trunk.html, result_details_4.1.tar.gz, result_details_5.0.tar.gz,
> result_details_trunk.tar.gz
>
>
> Observed instances of timeouts on clusters with long uptime and infrequently
> used tables and possibly just request paths such as not using CAS for large
> fractions of a year.
> CAS seems to be more severely impacted because it has more metrics in the
> request path such as latency measurements for prepare, propose, and the read
> from the underlying table.
> Tracing showed ~600-800 milliseconds for these operations in between the
> “appending to memtable” and “sending a response” events. Reads had a delay
> between finishing the construction of the iterator and sending the read
> response.
> Stack traces dumped every 100 milliseconds using {{sjk}} shows that in
> prepare and propose a lot of time was being spent in
> {{{}Meter.tickIfNecessary{}}}.
> {code:java}
> Thread [2537] RUNNABLE at 2024-01-25T21:14:48.218 - MutationStage-2
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:71)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.prepare(PaxosState.java:92)
> Thread [2539] RUNNABLE at 2024-01-25T21:14:48.520 - MutationStage-4
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:72)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.propose(PaxosState.java:127){code}
> {{tickIfNecessary}} does a linear amount of work proportional to the time
> since the last time the metric was updated/read/created and this can actually
> take a measurable amount of time even in a tight loop. On my M2 MBP it was
> 1.5 milliseconds for a day, ~200 days took ~74 milliseconds. Before it warmed
> up it was 140 milliseconds.
> A quick fix is to schedule a task to read all the meters once a day so it
> isn’t done in the request path and we have a more incremental amount to
> process at a time.
> Also observed that {{tickIfNecessary}} is not 100% thread safe in that if it
> takes longer than 5 seconds to run the loop it can end up with multiple
> threads attempting to run the loop at once and then they will concurrently
> run {{EWMA.tick}} which probably results in some ticks not being performed.
> This issue is still present in the latest version of {{Metrics}} if using
> {{{}EWMA{}}}, but {{SlidingWindowTimeAverages}} looks like it has a bounded
> amount of work required to tick. Switching would change how our metrics work
> since the two don't have the same behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-19332) Dropwizard Meter causes timeouts when infrequently used
[
https://issues.apache.org/jira/browse/CASSANDRA-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830554#comment-17830554
]
Stefan Miklosovic edited comment on CASSANDRA-19332 at 3/25/24 4:09 PM:
Too bad the PR for Dropwizard was not merged and released yet. Do I understand
it correctly that we do not need to have Dropwizard's PR merged because we
workaround it but this change will become not necessary anymore if PR is merged
and we upgrade the library?
was (Author: smiklosovic):
Too bad the PR for Dropwizard was not merge and released yet. Do I understand
it correctly that we do not need to have Dropwizard's PR merge because we
workaround it but this change will become not necessary anymore if PR is merged
and we upgrade the library?
> Dropwizard Meter causes timeouts when infrequently used
> ---
>
> Key: CASSANDRA-19332
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19332
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0, 5.1
>
> Attachments: ci_summary_4.1.html, ci_summary_5.0.html,
> ci_summary_trunk.html, result_details_4.1.tar.gz, result_details_5.0.tar.gz,
> result_details_trunk.tar.gz
>
>
> Observed instances of timeouts on clusters with long uptime and infrequently
> used tables and possibly just request paths such as not using CAS for large
> fractions of a year.
> CAS seems to be more severely impacted because it has more metrics in the
> request path such as latency measurements for prepare, propose, and the read
> from the underlying table.
> Tracing showed ~600-800 milliseconds for these operations in between the
> “appending to memtable” and “sending a response” events. Reads had a delay
> between finishing the construction of the iterator and sending the read
> response.
> Stack traces dumped every 100 milliseconds using {{sjk}} shows that in
> prepare and propose a lot of time was being spent in
> {{{}Meter.tickIfNecessary{}}}.
> {code:java}
> Thread [2537] RUNNABLE at 2024-01-25T21:14:48.218 - MutationStage-2
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:71)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.prepare(PaxosState.java:92)
> Thread [2539] RUNNABLE at 2024-01-25T21:14:48.520 - MutationStage-4
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:72)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.propose(PaxosState.java:127){code}
> {{tickIfNecessary}} does a linear amount of work proportional to the time
> since the last time the metric was updated/read/created and this can actually
> take a measurable amount of time even in a tight loop. On my M2 MBP it was
> 1.5 milliseconds for a day, ~200 days took ~74 milliseconds. Before it warmed
> up it was 140 milliseconds.
> A quick fix is to schedule a task to read all the meters once a day so it
> isn’t done in the request path and we have a more incremental amount to
> process at a time.
> Also observed that {{tickIfNecessary}} is not 100% thread safe in that if it
> takes longer than 5 seconds to run the loop it can end up with multiple
> threads attempting to run the loop at once and then they will concurrently
> run {{EWMA.tick}} which probably results in some ticks not being performed.
> This issue is still present in the latest version of {{Metrics}} if using
> {{{}EWMA{}}}, but {{SlidingWindowTimeAverages}} looks like it has a bounded
> amount of work required to tick. Switching would change how our metrics work
> since the two don't have the same behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19332) Dropwizard Meter causes timeouts when infrequently used
[
https://issues.apache.org/jira/browse/CASSANDRA-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830554#comment-17830554
]
Stefan Miklosovic commented on CASSANDRA-19332:
---
Too bad the PR for Dropwizard was not merge and released yet. Do I understand
it correctly that we do not need to have Dropwizard's PR merge because we
workaround it but this change will become not necessary anymore if PR is merged
and we upgrade the library?
> Dropwizard Meter causes timeouts when infrequently used
> ---
>
> Key: CASSANDRA-19332
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19332
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0, 5.1
>
> Attachments: ci_summary_4.1.html, ci_summary_5.0.html,
> ci_summary_trunk.html, result_details_4.1.tar.gz, result_details_5.0.tar.gz,
> result_details_trunk.tar.gz
>
>
> Observed instances of timeouts on clusters with long uptime and infrequently
> used tables and possibly just request paths such as not using CAS for large
> fractions of a year.
> CAS seems to be more severely impacted because it has more metrics in the
> request path such as latency measurements for prepare, propose, and the read
> from the underlying table.
> Tracing showed ~600-800 milliseconds for these operations in between the
> “appending to memtable” and “sending a response” events. Reads had a delay
> between finishing the construction of the iterator and sending the read
> response.
> Stack traces dumped every 100 milliseconds using {{sjk}} shows that in
> prepare and propose a lot of time was being spent in
> {{{}Meter.tickIfNecessary{}}}.
> {code:java}
> Thread [2537] RUNNABLE at 2024-01-25T21:14:48.218 - MutationStage-2
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:71)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.prepare(PaxosState.java:92)
> Thread [2539] RUNNABLE at 2024-01-25T21:14:48.520 - MutationStage-4
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:72)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.propose(PaxosState.java:127){code}
> {{tickIfNecessary}} does a linear amount of work proportional to the time
> since the last time the metric was updated/read/created and this can actually
> take a measurable amount of time even in a tight loop. On my M2 MBP it was
> 1.5 milliseconds for a day, ~200 days took ~74 milliseconds. Before it warmed
> up it was 140 milliseconds.
> A quick fix is to schedule a task to read all the meters once a day so it
> isn’t done in the request path and we have a more incremental amount to
> process at a time.
> Also observed that {{tickIfNecessary}} is not 100% thread safe in that if it
> takes longer than 5 seconds to run the loop it can end up with multiple
> threads attempting to run the loop at once and then they will concurrently
> run {{EWMA.tick}} which probably results in some ticks not being performed.
> This issue is still present in the latest version of {{Metrics}} if using
> {{{}EWMA{}}}, but {{SlidingWindowTimeAverages}} looks like it has a bounded
> amount of work required to tick. Switching would change how our metrics work
> since the two don't have the same behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19332) Dropwizard Meter causes timeouts when infrequently used
[
https://issues.apache.org/jira/browse/CASSANDRA-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830549#comment-17830549
]
Ariel Weisberg commented on CASSANDRA-19332:
I haven't merged this for a while because I have had a steady stream of CI
issues. Based on comparisons with nightlies it doesn't look like the test
failures are related. 5.0 is basically clean, 4.0/4.1 have a bunch of upgrade
test issues, and so does trunk. I think this is more specific to our CI
environment and some issues invoking different upgrade paths.
I'm going to go ahead and merge anyways unless someone objects since the scope
of this change is pretty narrow and doesn't look related to any of the failures.
> Dropwizard Meter causes timeouts when infrequently used
> ---
>
> Key: CASSANDRA-19332
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19332
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0, 5.1
>
> Attachments: ci_summary_4.1.html, ci_summary_5.0.html,
> ci_summary_trunk.html, result_details_4.1.tar.gz, result_details_5.0.tar.gz,
> result_details_trunk.tar.gz
>
>
> Observed instances of timeouts on clusters with long uptime and infrequently
> used tables and possibly just request paths such as not using CAS for large
> fractions of a year.
> CAS seems to be more severely impacted because it has more metrics in the
> request path such as latency measurements for prepare, propose, and the read
> from the underlying table.
> Tracing showed ~600-800 milliseconds for these operations in between the
> “appending to memtable” and “sending a response” events. Reads had a delay
> between finishing the construction of the iterator and sending the read
> response.
> Stack traces dumped every 100 milliseconds using {{sjk}} shows that in
> prepare and propose a lot of time was being spent in
> {{{}Meter.tickIfNecessary{}}}.
> {code:java}
> Thread [2537] RUNNABLE at 2024-01-25T21:14:48.218 - MutationStage-2
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:71)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.prepare(PaxosState.java:92)
> Thread [2539] RUNNABLE at 2024-01-25T21:14:48.520 - MutationStage-4
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:72)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.propose(PaxosState.java:127){code}
> {{tickIfNecessary}} does a linear amount of work proportional to the time
> since the last time the metric was updated/read/created and this can actually
> take a measurable amount of time even in a tight loop. On my M2 MBP it was
> 1.5 milliseconds for a day, ~200 days took ~74 milliseconds. Before it warmed
> up it was 140 milliseconds.
> A quick fix is to schedule a task to read all the meters once a day so it
> isn’t done in the request path and we have a more incremental amount to
> process at a time.
> Also observed that {{tickIfNecessary}} is not 100% thread safe in that if it
> takes longer than 5 seconds to run the loop it can end up with multiple
> threads attempting to run the loop at once and then they will concurrently
> run {{EWMA.tick}} which probably results in some ticks not being performed.
> This issue is still present in the latest version of {{Metrics}} if using
> {{{}EWMA{}}}, but {{SlidingWindowTimeAverages}} looks like it has a bounded
> amount of work required to tick. Switching would change how our metrics work
> since the two don't have the same behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19332) Dropwizard Meter causes timeouts when infrequently used
[
https://issues.apache.org/jira/browse/CASSANDRA-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ariel Weisberg updated CASSANDRA-19332:
---
Fix Version/s: 5.0
5.1
> Dropwizard Meter causes timeouts when infrequently used
> ---
>
> Key: CASSANDRA-19332
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19332
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0, 5.1
>
> Attachments: ci_summary_4.1.html, ci_summary_5.0.html,
> ci_summary_trunk.html, result_details_4.1.tar.gz, result_details_5.0.tar.gz,
> result_details_trunk.tar.gz
>
>
> Observed instances of timeouts on clusters with long uptime and infrequently
> used tables and possibly just request paths such as not using CAS for large
> fractions of a year.
> CAS seems to be more severely impacted because it has more metrics in the
> request path such as latency measurements for prepare, propose, and the read
> from the underlying table.
> Tracing showed ~600-800 milliseconds for these operations in between the
> “appending to memtable” and “sending a response” events. Reads had a delay
> between finishing the construction of the iterator and sending the read
> response.
> Stack traces dumped every 100 milliseconds using {{sjk}} shows that in
> prepare and propose a lot of time was being spent in
> {{{}Meter.tickIfNecessary{}}}.
> {code:java}
> Thread [2537] RUNNABLE at 2024-01-25T21:14:48.218 - MutationStage-2
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:71)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.prepare(PaxosState.java:92)
> Thread [2539] RUNNABLE at 2024-01-25T21:14:48.520 - MutationStage-4
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:72)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.propose(PaxosState.java:127){code}
> {{tickIfNecessary}} does a linear amount of work proportional to the time
> since the last time the metric was updated/read/created and this can actually
> take a measurable amount of time even in a tight loop. On my M2 MBP it was
> 1.5 milliseconds for a day, ~200 days took ~74 milliseconds. Before it warmed
> up it was 140 milliseconds.
> A quick fix is to schedule a task to read all the meters once a day so it
> isn’t done in the request path and we have a more incremental amount to
> process at a time.
> Also observed that {{tickIfNecessary}} is not 100% thread safe in that if it
> takes longer than 5 seconds to run the loop it can end up with multiple
> threads attempting to run the loop at once and then they will concurrently
> run {{EWMA.tick}} which probably results in some ticks not being performed.
> This issue is still present in the latest version of {{Metrics}} if using
> {{{}EWMA{}}}, but {{SlidingWindowTimeAverages}} looks like it has a bounded
> amount of work required to tick. Switching would change how our metrics work
> since the two don't have the same behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19332) Dropwizard Meter causes timeouts when infrequently used
[
https://issues.apache.org/jira/browse/CASSANDRA-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ariel Weisberg updated CASSANDRA-19332:
---
Attachment: ci_summary_4.1.html
result_details_4.1.tar.gz
> Dropwizard Meter causes timeouts when infrequently used
> ---
>
> Key: CASSANDRA-19332
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19332
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Normal
> Fix For: 4.0.x, 4.1.x
>
> Attachments: ci_summary_4.1.html, ci_summary_5.0.html,
> ci_summary_trunk.html, result_details_4.1.tar.gz, result_details_5.0.tar.gz,
> result_details_trunk.tar.gz
>
>
> Observed instances of timeouts on clusters with long uptime and infrequently
> used tables and possibly just request paths such as not using CAS for large
> fractions of a year.
> CAS seems to be more severely impacted because it has more metrics in the
> request path such as latency measurements for prepare, propose, and the read
> from the underlying table.
> Tracing showed ~600-800 milliseconds for these operations in between the
> “appending to memtable” and “sending a response” events. Reads had a delay
> between finishing the construction of the iterator and sending the read
> response.
> Stack traces dumped every 100 milliseconds using {{sjk}} shows that in
> prepare and propose a lot of time was being spent in
> {{{}Meter.tickIfNecessary{}}}.
> {code:java}
> Thread [2537] RUNNABLE at 2024-01-25T21:14:48.218 - MutationStage-2
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:71)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.prepare(PaxosState.java:92)
> Thread [2539] RUNNABLE at 2024-01-25T21:14:48.520 - MutationStage-4
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:72)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.propose(PaxosState.java:127){code}
> {{tickIfNecessary}} does a linear amount of work proportional to the time
> since the last time the metric was updated/read/created and this can actually
> take a measurable amount of time even in a tight loop. On my M2 MBP it was
> 1.5 milliseconds for a day, ~200 days took ~74 milliseconds. Before it warmed
> up it was 140 milliseconds.
> A quick fix is to schedule a task to read all the meters once a day so it
> isn’t done in the request path and we have a more incremental amount to
> process at a time.
> Also observed that {{tickIfNecessary}} is not 100% thread safe in that if it
> takes longer than 5 seconds to run the loop it can end up with multiple
> threads attempting to run the loop at once and then they will concurrently
> run {{EWMA.tick}} which probably results in some ticks not being performed.
> This issue is still present in the latest version of {{Metrics}} if using
> {{{}EWMA{}}}, but {{SlidingWindowTimeAverages}} looks like it has a bounded
> amount of work required to tick. Switching would change how our metrics work
> since the two don't have the same behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19486) Add method for sizes of hints and hints per node in HintsServiceMBean and enrich system_views.pending_hints with sizes
[ https://issues.apache.org/jira/browse/CASSANDRA-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Miklosovic updated CASSANDRA-19486: -- Summary: Add method for sizes of hints and hints per node in HintsServiceMBean and enrich system_views.pending_hints with sizes (was: Add method for sizes of hints and hints per node in HintsServiceMBean) > Add method for sizes of hints and hints per node in HintsServiceMBean and > enrich system_views.pending_hints with sizes > -- > > Key: CASSANDRA-19486 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19486 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Hints >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 5.x > > Time Spent: 50m > Remaining Estimate: 0h > > I am happy to close this if somebody shows me how to get total size of all > hints or all hints per particular node via JMX. > I could find StorageMetrics.totalHints but that is how many hints there are, > not their sizes. We also have > org.apache.cassandra.metrics:type=HintedHandOffManager name= > org.apache.cassandra.metrics:type=HintsService name= > But that is again showing other metrics not sizes. > I would add two methods into HintsServiceMBean returning this. Seems to be > very easy to do once we do CASSANDRA-19477. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19486) Add method for sizes of hints and hints per node in HintsServiceMBean
[ https://issues.apache.org/jira/browse/CASSANDRA-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830543#comment-17830543 ] Stefan Miklosovic commented on CASSANDRA-19486: --- It will be better if we leave nodetool behind and put this only to CQL vtable system_views.pending_hints. Also, I think that having two methods on JMX MBean wont kill anybody, we can still have JMX way to query that but not on nodetool for some emergency situations / custom admin jmx tooling. > Add method for sizes of hints and hints per node in HintsServiceMBean > - > > Key: CASSANDRA-19486 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19486 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Hints >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 5.x > > Time Spent: 50m > Remaining Estimate: 0h > > I am happy to close this if somebody shows me how to get total size of all > hints or all hints per particular node via JMX. > I could find StorageMetrics.totalHints but that is how many hints there are, > not their sizes. We also have > org.apache.cassandra.metrics:type=HintedHandOffManager name= > org.apache.cassandra.metrics:type=HintsService name= > But that is again showing other metrics not sizes. > I would add two methods into HintsServiceMBean returning this. Seems to be > very easy to do once we do CASSANDRA-19477. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19332) Dropwizard Meter causes timeouts when infrequently used
[
https://issues.apache.org/jira/browse/CASSANDRA-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ariel Weisberg updated CASSANDRA-19332:
---
Attachment: ci_summary_5.0.html
result_details_5.0.tar.gz
> Dropwizard Meter causes timeouts when infrequently used
> ---
>
> Key: CASSANDRA-19332
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19332
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Normal
> Fix For: 4.0.x, 4.1.x
>
> Attachments: ci_summary_5.0.html, ci_summary_trunk.html,
> result_details_5.0.tar.gz, result_details_trunk.tar.gz
>
>
> Observed instances of timeouts on clusters with long uptime and infrequently
> used tables and possibly just request paths such as not using CAS for large
> fractions of a year.
> CAS seems to be more severely impacted because it has more metrics in the
> request path such as latency measurements for prepare, propose, and the read
> from the underlying table.
> Tracing showed ~600-800 milliseconds for these operations in between the
> “appending to memtable” and “sending a response” events. Reads had a delay
> between finishing the construction of the iterator and sending the read
> response.
> Stack traces dumped every 100 milliseconds using {{sjk}} shows that in
> prepare and propose a lot of time was being spent in
> {{{}Meter.tickIfNecessary{}}}.
> {code:java}
> Thread [2537] RUNNABLE at 2024-01-25T21:14:48.218 - MutationStage-2
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:71)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.prepare(PaxosState.java:92)
> Thread [2539] RUNNABLE at 2024-01-25T21:14:48.520 - MutationStage-4
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:72)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.propose(PaxosState.java:127){code}
> {{tickIfNecessary}} does a linear amount of work proportional to the time
> since the last time the metric was updated/read/created and this can actually
> take a measurable amount of time even in a tight loop. On my M2 MBP it was
> 1.5 milliseconds for a day, ~200 days took ~74 milliseconds. Before it warmed
> up it was 140 milliseconds.
> A quick fix is to schedule a task to read all the meters once a day so it
> isn’t done in the request path and we have a more incremental amount to
> process at a time.
> Also observed that {{tickIfNecessary}} is not 100% thread safe in that if it
> takes longer than 5 seconds to run the loop it can end up with multiple
> threads attempting to run the loop at once and then they will concurrently
> run {{EWMA.tick}} which probably results in some ticks not being performed.
> This issue is still present in the latest version of {{Metrics}} if using
> {{{}EWMA{}}}, but {{SlidingWindowTimeAverages}} looks like it has a bounded
> amount of work required to tick. Switching would change how our metrics work
> since the two don't have the same behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19332) Dropwizard Meter causes timeouts when infrequently used
[
https://issues.apache.org/jira/browse/CASSANDRA-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ariel Weisberg updated CASSANDRA-19332:
---
Attachment: ci_summary_trunk.html
result_details_trunk.tar.gz
> Dropwizard Meter causes timeouts when infrequently used
> ---
>
> Key: CASSANDRA-19332
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19332
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Normal
> Fix For: 4.0.x, 4.1.x
>
> Attachments: ci_summary_trunk.html, result_details_trunk.tar.gz
>
>
> Observed instances of timeouts on clusters with long uptime and infrequently
> used tables and possibly just request paths such as not using CAS for large
> fractions of a year.
> CAS seems to be more severely impacted because it has more metrics in the
> request path such as latency measurements for prepare, propose, and the read
> from the underlying table.
> Tracing showed ~600-800 milliseconds for these operations in between the
> “appending to memtable” and “sending a response” events. Reads had a delay
> between finishing the construction of the iterator and sending the read
> response.
> Stack traces dumped every 100 milliseconds using {{sjk}} shows that in
> prepare and propose a lot of time was being spent in
> {{{}Meter.tickIfNecessary{}}}.
> {code:java}
> Thread [2537] RUNNABLE at 2024-01-25T21:14:48.218 - MutationStage-2
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:71)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.prepare(PaxosState.java:92)
> Thread [2539] RUNNABLE at 2024-01-25T21:14:48.520 - MutationStage-4
> com.codahale.metrics.Meter.tickIfNecessary(Meter.java:72)
> com.codahale.metrics.Meter.mark(Meter.java:55)
> com.codahale.metrics.Meter.mark(Meter.java:46)
> com.codahale.metrics.Timer.update(Timer.java:150)
> com.codahale.metrics.Timer.update(Timer.java:86)
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
> org.apache.cassandra.service.paxos.PaxosState.propose(PaxosState.java:127){code}
> {{tickIfNecessary}} does a linear amount of work proportional to the time
> since the last time the metric was updated/read/created and this can actually
> take a measurable amount of time even in a tight loop. On my M2 MBP it was
> 1.5 milliseconds for a day, ~200 days took ~74 milliseconds. Before it warmed
> up it was 140 milliseconds.
> A quick fix is to schedule a task to read all the meters once a day so it
> isn’t done in the request path and we have a more incremental amount to
> process at a time.
> Also observed that {{tickIfNecessary}} is not 100% thread safe in that if it
> takes longer than 5 seconds to run the loop it can end up with multiple
> threads attempting to run the loop at once and then they will concurrently
> run {{EWMA.tick}} which probably results in some ticks not being performed.
> This issue is still present in the latest version of {{Metrics}} if using
> {{{}EWMA{}}}, but {{SlidingWindowTimeAverages}} looks like it has a bounded
> amount of work required to tick. Switching would change how our metrics work
> since the two don't have the same behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19182) IR may leak SSTables with pending repair when coming from streaming
[ https://issues.apache.org/jira/browse/CASSANDRA-19182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830531#comment-17830531 ] Brandon Williams commented on CASSANDRA-19182: -- If I push 4.0 through circle will that help? > IR may leak SSTables with pending repair when coming from streaming > --- > > Key: CASSANDRA-19182 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19182 > Project: Cassandra > Issue Type: Bug > Components: Consistency/Repair >Reporter: David Capwell >Assignee: David Capwell >Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Attachments: > ci_summary-trunk-a1010f4101bf259de3f31077540e4f987d5df9c5.html > > Time Spent: 1h 40m > Remaining Estimate: 0h > > There is a race condition where SSTables from streaming may race with pending > repair cleanup in compaction causing us to cleanup the pending repair state > in compaction while the SSTables are being added to it; this leads to IR > failing in the future when those files get selected for repair. > This problem was hard to track down as the in-memory state was wiped, so we > don’t have any details. To better aid these types of investigation we should > make sure the repair vtables get updated when IR session failures are > submitted -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19486) Add method for sizes of hints and hints per node in HintsServiceMBean
[ https://issues.apache.org/jira/browse/CASSANDRA-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Miklosovic updated CASSANDRA-19486: -- Test and Documentation Plan: ci Status: Patch Available (was: In Progress) > Add method for sizes of hints and hints per node in HintsServiceMBean > - > > Key: CASSANDRA-19486 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19486 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Hints >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 5.x > > Time Spent: 20m > Remaining Estimate: 0h > > I am happy to close this if somebody shows me how to get total size of all > hints or all hints per particular node via JMX. > I could find StorageMetrics.totalHints but that is how many hints there are, > not their sizes. We also have > org.apache.cassandra.metrics:type=HintedHandOffManager name= > org.apache.cassandra.metrics:type=HintsService name= > But that is again showing other metrics not sizes. > I would add two methods into HintsServiceMBean returning this. Seems to be > very easy to do once we do CASSANDRA-19477. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19486) Add method for sizes of hints and hints per node in HintsServiceMBean
[ https://issues.apache.org/jira/browse/CASSANDRA-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Miklosovic updated CASSANDRA-19486: -- Status: Needs Committer (was: Patch Available) > Add method for sizes of hints and hints per node in HintsServiceMBean > - > > Key: CASSANDRA-19486 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19486 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Hints >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 5.x > > Time Spent: 20m > Remaining Estimate: 0h > > I am happy to close this if somebody shows me how to get total size of all > hints or all hints per particular node via JMX. > I could find StorageMetrics.totalHints but that is how many hints there are, > not their sizes. We also have > org.apache.cassandra.metrics:type=HintedHandOffManager name= > org.apache.cassandra.metrics:type=HintsService name= > But that is again showing other metrics not sizes. > I would add two methods into HintsServiceMBean returning this. Seems to be > very easy to do once we do CASSANDRA-19477. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19486) Add method for sizes of hints and hints per node in HintsServiceMBean
[ https://issues.apache.org/jira/browse/CASSANDRA-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Miklosovic updated CASSANDRA-19486: -- Change Category: Operability Complexity: Low Hanging Fruit Status: Open (was: Triage Needed) > Add method for sizes of hints and hints per node in HintsServiceMBean > - > > Key: CASSANDRA-19486 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19486 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Hints >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 5.x > > Time Spent: 20m > Remaining Estimate: 0h > > I am happy to close this if somebody shows me how to get total size of all > hints or all hints per particular node via JMX. > I could find StorageMetrics.totalHints but that is how many hints there are, > not their sizes. We also have > org.apache.cassandra.metrics:type=HintedHandOffManager name= > org.apache.cassandra.metrics:type=HintsService name= > But that is again showing other metrics not sizes. > I would add two methods into HintsServiceMBean returning this. Seems to be > very easy to do once we do CASSANDRA-19477. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19485) if max_hints_size_per_host < max_hints_file_size then it will write hints after max_hints_size_per_host is reached
[
https://issues.apache.org/jira/browse/CASSANDRA-19485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic updated CASSANDRA-19485:
--
Fix Version/s: (was: 5.x)
(was: 4.1.x)
(was: 5.0.x)
> if max_hints_size_per_host < max_hints_file_size then it will write hints
> after max_hints_size_per_host is reached
> --
>
> Key: CASSANDRA-19485
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19485
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Hints
>Reporter: Stefan Miklosovic
>Priority: Normal
>
> there is one problem in the current solution being that if we have this config
> {noformat}
> max_hints_size_per_host: 2MiB
> max_hints_file_size: 128MiB
> {noformat}
> basically, max size > size per host, then it will not stop it from writing
> hints after 2MiB for a particular node, because HintsDescriptor is added
> among dispatching ones after writer is closed, which happens after there is
> 128MiB written (all logic in HintsStore), so it will not be included into
> total sizes, it will be 0 until at least one hints file for that node is
> written to disk.
> I consider this to be the flaw of CASSANDRA-17142, however it is questionable
> if this is serious enough to deal with in the first place, I don't think
> somebody would set it up in practice to these values, normally one puts there
> like max per host is few gigs so this problem is not so visible but it shows
> in tests almost instantly and it is technically just wrong, regardless of the
> probability this would happen in real ...
> cc [~yifanc]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19485) if max_hints_size_per_host < max_hints_file_size then it will write hints after max_hints_size_per_host is reached
[
https://issues.apache.org/jira/browse/CASSANDRA-19485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic updated CASSANDRA-19485:
--
Resolution: Resolved
Status: Resolved (was: Triage Needed)
The fix for this is done as part of CASSANDRA-19477
> if max_hints_size_per_host < max_hints_file_size then it will write hints
> after max_hints_size_per_host is reached
> --
>
> Key: CASSANDRA-19485
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19485
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Hints
>Reporter: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.1.x, 5.0.x, 5.x
>
>
> there is one problem in the current solution being that if we have this config
> {noformat}
> max_hints_size_per_host: 2MiB
> max_hints_file_size: 128MiB
> {noformat}
> basically, max size > size per host, then it will not stop it from writing
> hints after 2MiB for a particular node, because HintsDescriptor is added
> among dispatching ones after writer is closed, which happens after there is
> 128MiB written (all logic in HintsStore), so it will not be included into
> total sizes, it will be 0 until at least one hints file for that node is
> written to disk.
> I consider this to be the flaw of CASSANDRA-17142, however it is questionable
> if this is serious enough to deal with in the first place, I don't think
> somebody would set it up in practice to these values, normally one puts there
> like max per host is few gigs so this problem is not so visible but it shows
> in tests almost instantly and it is technically just wrong, regardless of the
> probability this would happen in real ...
> cc [~yifanc]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19477) Do not go to disk to get HintsStore.getTotalFileSize
[
https://issues.apache.org/jira/browse/CASSANDRA-19477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic updated CASSANDRA-19477:
--
Fix Version/s: 4.1.5
5.0-beta2
5.1-alpha1
(was: 5.x)
(was: 4.1.x)
(was: 5.0-rc)
Since Version: 4.1-alpha1
Source Control Link:
https://github.com/apache/cassandra/commit/38408938ccfe5b8c051e25c645bdcd71b45fa66e
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
> Do not go to disk to get HintsStore.getTotalFileSize
>
>
> Key: CASSANDRA-19477
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19477
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Hints
>Reporter: Jon Haddad
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.1.5, 5.0-beta2, 5.1-alpha1
>
> Attachments: flame-cassandra0-patched-2024-03-25_00-40-47.html,
> flame-cassandra0-release-2024-03-25_00-16-44.html, flamegraph.cpu.html,
> image-2024-03-24-17-57-32-560.png, image-2024-03-24-18-08-36-918.png,
> image-2024-03-24-18-16-50-370.png, image-2024-03-24-18-17-48-334.png,
> image-2024-03-24-18-20-07-734.png
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> When testing a cluster with more requests than it could handle, I noticed
> significant CPU time (25%) spent in HintsStore.getTotalFileSize. Here's what
> I'm seeing from profiling:
> 10% of CPU time spent in HintsDescriptor.fileName which only does this:
>
> {noformat}
> return String.format("%s-%s-%s.hints", hostId, timestamp, version);{noformat}
> At a bare minimum here we should create this string up front with the host
> and version and eliminate 2 of the 3 substitutions, but I think it's probably
> faster to use a StringBuilder and avoid the underlying regular expression
> altogether.
> 12% of the time is spent in org.apache.cassandra.io.util.File.length. It
> looks like this is called once for each hint file on disk for each host we're
> hinting to. In the case of an overloaded cluster, this is significant. It
> would be better if we were to track the file size in memory for each hint
> file and reference that rather than go to the filesystem.
> These fairly small changes should make Cassandra more reliable when under
> load spikes.
> CPU Flame graph attached.
> I only tested this in 4.1 but it looks like this is present up to trunk.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) 01/01: Merge branch 'cassandra-5.0' into trunk
This is an automated email from the ASF dual-hosted git repository.
smiklosovic pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit e6a937c934bf9aa738dfa8d6daa4ced48daf8859
Merge: 8d8c6fbc37 35d6e6ce9c
Author: Stefan Miklosovic
AuthorDate: Mon Mar 25 14:21:25 2024 +0100
Merge branch 'cassandra-5.0' into trunk
CHANGES.txt| 1 +
.../apache/cassandra/hints/HintsDescriptor.java| 24 +++-
.../org/apache/cassandra/hints/HintsStore.java | 12 +-
.../org/apache/cassandra/hints/HintsWriter.java| 13 ++-
.../org/apache/cassandra/service/StorageProxy.java | 14 ++-
.../distributed/test/HintsMaxSizeTest.java | 128 +
.../org/apache/cassandra/hints/AlteredHints.java | 3 +
.../apache/cassandra/hints/HintsCatalogTest.java | 17 +--
.../apache/cassandra/hints/HintsReaderTest.java| 5 +
.../org/apache/cassandra/hints/HintsStoreTest.java | 3 +
.../apache/cassandra/hints/HintsUpgradeTest.java | 3 +
11 files changed, 197 insertions(+), 26 deletions(-)
diff --cc CHANGES.txt
index e92cd81496,e879141be7..5a0ce4117d
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -54,9 -28,10 +54,10 @@@ Merged from 5.0
* Fix resource cleanup after SAI query timeouts (CASSANDRA-19177)
* Suppress CVE-2023-6481 (CASSANDRA-19184)
Merged from 4.1:
+ * Do not go to disk for reading hints file sizes (CASSANDRA-19477)
* Fix system_views.settings to handle array types (CASSANDRA-19475)
- * Memoize Cassandra verion and add a backoff interval for failed schema
pulls (CASSANDRA-18902)
* Fix StackOverflowError on ALTER after many previous schema changes
(CASSANDRA-19166)
+ * Memoize Cassandra verion (CASSANDRA-18902)
Merged from 4.0:
* Push LocalSessions info logs to debug (CASSANDRA-18335)
* Filter remote DC replicas out when constructing the initial replica plan
for the local read repair (CASSANDRA-19120)
diff --cc test/unit/org/apache/cassandra/hints/HintsReaderTest.java
index 77c45c6a12,56154fd950..94be7b2b8e
--- a/test/unit/org/apache/cassandra/hints/HintsReaderTest.java
+++ b/test/unit/org/apache/cassandra/hints/HintsReaderTest.java
@@@ -28,9 -28,9 +28,11 @@@ import java.util.concurrent.TimeUnit
import java.util.function.Function;
import com.google.common.collect.Iterables;
+
+import org.apache.cassandra.exceptions.CoordinatorBehindException;
import org.apache.cassandra.io.util.File;
+
+ import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.Test;
diff --cc test/unit/org/apache/cassandra/hints/HintsUpgradeTest.java
index 3d8c443121,78b8f56b50..fd44a96317
--- a/test/unit/org/apache/cassandra/hints/HintsUpgradeTest.java
+++ b/test/unit/org/apache/cassandra/hints/HintsUpgradeTest.java
@@@ -44,11 -43,12 +44,13 @@@ import org.apache.cassandra.schema.Sche
import org.apache.cassandra.schema.TableId;
import org.apache.cassandra.schema.TableMetadata;
+ import static org.hamcrest.Matchers.greaterThan;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
+ import static org.junit.Assert.assertThat;
import static org.junit.Assert.assertTrue;
+@Ignore("TODO: TCM")
public class HintsUpgradeTest
{
static
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch trunk updated (8d8c6fbc37 -> e6a937c934)
This is an automated email from the ASF dual-hosted git repository. smiklosovic pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git from 8d8c6fbc37 Override the correct method to avoid retries in ConsistentBootstrapTest.coordinatorIsBehindTest add 38408938cc Do not go to disk for reading hints file sizes add 35d6e6ce9c Merge branch 'cassandra-4.1' into cassandra-5.0 new e6a937c934 Merge branch 'cassandra-5.0' into trunk The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 1 + .../apache/cassandra/hints/HintsDescriptor.java| 24 +++- .../org/apache/cassandra/hints/HintsStore.java | 12 +- .../org/apache/cassandra/hints/HintsWriter.java| 13 ++- .../org/apache/cassandra/service/StorageProxy.java | 14 ++- .../distributed/test/HintsMaxSizeTest.java | 128 + .../org/apache/cassandra/hints/AlteredHints.java | 3 + .../apache/cassandra/hints/HintsCatalogTest.java | 17 +-- .../apache/cassandra/hints/HintsReaderTest.java| 5 + .../org/apache/cassandra/hints/HintsStoreTest.java | 3 + .../apache/cassandra/hints/HintsUpgradeTest.java | 3 + 11 files changed, 197 insertions(+), 26 deletions(-) create mode 100644 test/distributed/org/apache/cassandra/distributed/test/HintsMaxSizeTest.java - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch cassandra-5.0 updated (389479c9a2 -> 35d6e6ce9c)
This is an automated email from the ASF dual-hosted git repository. smiklosovic pushed a change to branch cassandra-5.0 in repository https://gitbox.apache.org/repos/asf/cassandra.git from 389479c9a2 Align buffer with commitlog segment size add 38408938cc Do not go to disk for reading hints file sizes add 35d6e6ce9c Merge branch 'cassandra-4.1' into cassandra-5.0 No new revisions were added by this update. Summary of changes: CHANGES.txt| 1 + .../apache/cassandra/hints/HintsDescriptor.java| 24 +++- .../org/apache/cassandra/hints/HintsStore.java | 12 +- .../org/apache/cassandra/hints/HintsWriter.java| 13 ++- .../org/apache/cassandra/service/StorageProxy.java | 14 ++- .../distributed/test/HintsMaxSizeTest.java | 128 + .../org/apache/cassandra/hints/AlteredHints.java | 3 + .../apache/cassandra/hints/HintsCatalogTest.java | 17 +-- .../apache/cassandra/hints/HintsReaderTest.java| 5 + .../org/apache/cassandra/hints/HintsStoreTest.java | 3 + .../apache/cassandra/hints/HintsUpgradeTest.java | 3 + 11 files changed, 197 insertions(+), 26 deletions(-) create mode 100644 test/distributed/org/apache/cassandra/distributed/test/HintsMaxSizeTest.java - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch cassandra-4.1 updated (96692d7206 -> 38408938cc)
This is an automated email from the ASF dual-hosted git repository. smiklosovic pushed a change to branch cassandra-4.1 in repository https://gitbox.apache.org/repos/asf/cassandra.git from 96692d7206 Merge branch 'cassandra-4.0' into cassandra-4.1 add 38408938cc Do not go to disk for reading hints file sizes No new revisions were added by this update. Summary of changes: CHANGES.txt| 1 + .../apache/cassandra/hints/HintsDescriptor.java| 24 +++- .../org/apache/cassandra/hints/HintsStore.java | 13 ++- .../org/apache/cassandra/hints/HintsWriter.java| 11 +- .../org/apache/cassandra/service/StorageProxy.java | 14 ++- .../distributed/test/HintsMaxSizeTest.java | 128 + .../org/apache/cassandra/hints/AlteredHints.java | 3 + .../apache/cassandra/hints/HintsCatalogTest.java | 17 +-- .../apache/cassandra/hints/HintsReaderTest.java| 5 + .../org/apache/cassandra/hints/HintsStoreTest.java | 3 + 10 files changed, 194 insertions(+), 25 deletions(-) create mode 100644 test/distributed/org/apache/cassandra/distributed/test/HintsMaxSizeTest.java - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19343) Test Failure: org.apache.cassandra.fuzz.ring.ConsistentBootstrapTest.coordinatorIsBehindTest
[
https://issues.apache.org/jira/browse/CASSANDRA-19343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-19343:
Fix Version/s: 5.1-alpha1
(was: 5.x)
Since Version: 5.x
Source Control Link:
https://github.com/apache/cassandra/commit/8d8c6fbc37899ff77be6b3431f99f6951c4c05c2
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
and committed, thanks!
> Test Failure:
> org.apache.cassandra.fuzz.ring.ConsistentBootstrapTest.coordinatorIsBehindTest
>
>
> Key: CASSANDRA-19343
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19343
> Project: Cassandra
> Issue Type: Bug
> Components: CI
>Reporter: Ekaterina Dimitrova
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 5.1-alpha1
>
> Attachments: ci_summary.html, result_details.tar.gz
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java}
> java.lang.IllegalStateException: Can't use shutdown instances, delegate is
> null at
> org.apache.cassandra.distributed.impl.AbstractCluster$Wrapper.delegate(AbstractCluster.java:283)
> at
> org.apache.cassandra.distributed.impl.DelegatingInvokableInstance.transfer(DelegatingInvokableInstance.java:49)
> at
> org.apache.cassandra.distributed.api.IInvokableInstance.runsOnInstance(IInvokableInstance.java:45)
> at
> org.apache.cassandra.distributed.api.IInvokableInstance.runOnInstance(IInvokableInstance.java:46)
> at
> org.apache.cassandra.distributed.shared.ClusterUtils.unpauseCommits(ClusterUtils.java:548)
> at
> org.apache.cassandra.fuzz.ring.ConsistentBootstrapTest.coordinatorIsBehindTest(ConsistentBootstrapTest.java:227)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
> https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2636/workflows/93adbf3e-acf8-4a62-a1f6-baf4f4689347/jobs/54912/tests
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19343) Test Failure: org.apache.cassandra.fuzz.ring.ConsistentBootstrapTest.coordinatorIsBehindTest
[
https://issues.apache.org/jira/browse/CASSANDRA-19343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-19343:
Status: Ready to Commit (was: Review In Progress)
> Test Failure:
> org.apache.cassandra.fuzz.ring.ConsistentBootstrapTest.coordinatorIsBehindTest
>
>
> Key: CASSANDRA-19343
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19343
> Project: Cassandra
> Issue Type: Bug
> Components: CI
>Reporter: Ekaterina Dimitrova
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 5.x
>
> Attachments: ci_summary.html, result_details.tar.gz
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java}
> java.lang.IllegalStateException: Can't use shutdown instances, delegate is
> null at
> org.apache.cassandra.distributed.impl.AbstractCluster$Wrapper.delegate(AbstractCluster.java:283)
> at
> org.apache.cassandra.distributed.impl.DelegatingInvokableInstance.transfer(DelegatingInvokableInstance.java:49)
> at
> org.apache.cassandra.distributed.api.IInvokableInstance.runsOnInstance(IInvokableInstance.java:45)
> at
> org.apache.cassandra.distributed.api.IInvokableInstance.runOnInstance(IInvokableInstance.java:46)
> at
> org.apache.cassandra.distributed.shared.ClusterUtils.unpauseCommits(ClusterUtils.java:548)
> at
> org.apache.cassandra.fuzz.ring.ConsistentBootstrapTest.coordinatorIsBehindTest(ConsistentBootstrapTest.java:227)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
> https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2636/workflows/93adbf3e-acf8-4a62-a1f6-baf4f4689347/jobs/54912/tests
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19343) Test Failure: org.apache.cassandra.fuzz.ring.ConsistentBootstrapTest.coordinatorIsBehindTest
[
https://issues.apache.org/jira/browse/CASSANDRA-19343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-19343:
Reviewers: Alex Petrov, Marcus Eriksson (was: Alex Petrov)
Status: Review In Progress (was: Patch Available)
> Test Failure:
> org.apache.cassandra.fuzz.ring.ConsistentBootstrapTest.coordinatorIsBehindTest
>
>
> Key: CASSANDRA-19343
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19343
> Project: Cassandra
> Issue Type: Bug
> Components: CI
>Reporter: Ekaterina Dimitrova
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 5.x
>
> Attachments: ci_summary.html, result_details.tar.gz
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java}
> java.lang.IllegalStateException: Can't use shutdown instances, delegate is
> null at
> org.apache.cassandra.distributed.impl.AbstractCluster$Wrapper.delegate(AbstractCluster.java:283)
> at
> org.apache.cassandra.distributed.impl.DelegatingInvokableInstance.transfer(DelegatingInvokableInstance.java:49)
> at
> org.apache.cassandra.distributed.api.IInvokableInstance.runsOnInstance(IInvokableInstance.java:45)
> at
> org.apache.cassandra.distributed.api.IInvokableInstance.runOnInstance(IInvokableInstance.java:46)
> at
> org.apache.cassandra.distributed.shared.ClusterUtils.unpauseCommits(ClusterUtils.java:548)
> at
> org.apache.cassandra.fuzz.ring.ConsistentBootstrapTest.coordinatorIsBehindTest(ConsistentBootstrapTest.java:227)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
> https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2636/workflows/93adbf3e-acf8-4a62-a1f6-baf4f4689347/jobs/54912/tests
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch trunk updated: Override the correct method to avoid retries in ConsistentBootstrapTest.coordinatorIsBehindTest
This is an automated email from the ASF dual-hosted git repository.
marcuse pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new 8d8c6fbc37 Override the correct method to avoid retries in
ConsistentBootstrapTest.coordinatorIsBehindTest
8d8c6fbc37 is described below
commit 8d8c6fbc37899ff77be6b3431f99f6951c4c05c2
Author: Marcus Eriksson
AuthorDate: Fri Mar 22 13:39:34 2024 +0100
Override the correct method to avoid retries in
ConsistentBootstrapTest.coordinatorIsBehindTest
Patch by marcuse; reviewed by Alex Petrov for CASSANDRA-19343
---
.../org/apache/cassandra/fuzz/ring/ConsistentBootstrapTest.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/test/distributed/org/apache/cassandra/fuzz/ring/ConsistentBootstrapTest.java
b/test/distributed/org/apache/cassandra/fuzz/ring/ConsistentBootstrapTest.java
index c03047478f..592d88be51 100644
---
a/test/distributed/org/apache/cassandra/fuzz/ring/ConsistentBootstrapTest.java
+++
b/test/distributed/org/apache/cassandra/fuzz/ring/ConsistentBootstrapTest.java
@@ -127,11 +127,11 @@ public class ConsistentBootstrapTest extends FuzzTestBase
ReplayingHistoryBuilder harry = HarryHelper.dataGen(new
InJvmSut(cluster, () -> 2, (t) -> false)
{
-public
Object[][] execute(String statement, ConsistencyLevel cl, int coordinator,
Object... bindings)
+public
Object[][] execute(String statement, ConsistencyLevel cl, int coordinator, int
pagesize, Object... bindings)
{
try
{
-
return super.execute(statement, cl, coordinator, bindings);
+
return super.execute(statement, cl, coordinator, pagesize, bindings);
}
catch
(Throwable t)
{
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19471) Commitlog with direct io fails test_change_durable_writes
[
https://issues.apache.org/jira/browse/CASSANDRA-19471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-19471:
-
Status: Ready to Commit (was: Review In Progress)
> Commitlog with direct io fails test_change_durable_writes
> -
>
> Key: CASSANDRA-19471
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19471
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Commit Log
>Reporter: Brandon Williams
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 5.0-rc, 5.x
>
>
> With the commitlog_disk_access_mode set to direct, and the improved
> configuration_test.py::TestConfiguration::test_change_durable_writes from
> CASSANDRA-19465, this fails with either:
> {noformat}
> AssertionError: Commitlog was written with durable writes disabled
> {noformat}
> Or what appears to be the original exception reported in CASSANDRA-19465:
> {noformat}
> node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,465
> StorageService.java:631 - Stopping native transport
> node1: ERROR [MutationStage-5] 2024-03-14 17:16:08,465
> StorageProxy.java:1670 - Failed to apply mutation locally :
> java.lang.IllegalArgumentException: newPosition > limit: (1048634 > 1048576)
> at java.base/java.nio.Buffer.createPositionException(Buffer.java:341)
> at java.base/java.nio.Buffer.position(Buffer.java:316)
> at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516)
> at
> java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:321)
> at
> java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:73)
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.allocate(CommitLogSegment.java:216)
> at
> org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard.allocate(CommitLogSegmentManagerStandard.java:52)
> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
> at
> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99)
> at
> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53)
> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:612)
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:497)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:244)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:264)
> at
> org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1664)
> at
> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2624)
> at
> org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:833)
> node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,470
> StorageService.java:636 - Stopping gossiper
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19471) Commitlog with direct io fails test_change_durable_writes
[
https://issues.apache.org/jira/browse/CASSANDRA-19471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-19471:
-
Reviewers: Berenguer Blasi, Brandon Williams (was: Berenguer Blasi)
Status: Review In Progress (was: Patch Available)
> Commitlog with direct io fails test_change_durable_writes
> -
>
> Key: CASSANDRA-19471
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19471
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Commit Log
>Reporter: Brandon Williams
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 5.0-rc, 5.x
>
>
> With the commitlog_disk_access_mode set to direct, and the improved
> configuration_test.py::TestConfiguration::test_change_durable_writes from
> CASSANDRA-19465, this fails with either:
> {noformat}
> AssertionError: Commitlog was written with durable writes disabled
> {noformat}
> Or what appears to be the original exception reported in CASSANDRA-19465:
> {noformat}
> node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,465
> StorageService.java:631 - Stopping native transport
> node1: ERROR [MutationStage-5] 2024-03-14 17:16:08,465
> StorageProxy.java:1670 - Failed to apply mutation locally :
> java.lang.IllegalArgumentException: newPosition > limit: (1048634 > 1048576)
> at java.base/java.nio.Buffer.createPositionException(Buffer.java:341)
> at java.base/java.nio.Buffer.position(Buffer.java:316)
> at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516)
> at
> java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:321)
> at
> java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:73)
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.allocate(CommitLogSegment.java:216)
> at
> org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard.allocate(CommitLogSegmentManagerStandard.java:52)
> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
> at
> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99)
> at
> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53)
> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:612)
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:497)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:244)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:264)
> at
> org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1664)
> at
> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2624)
> at
> org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:833)
> node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,470
> StorageService.java:636 - Stopping gossiper
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19471) Commitlog with direct io fails test_change_durable_writes
[
https://issues.apache.org/jira/browse/CASSANDRA-19471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-19471:
-
Fix Version/s: 5.0-beta2
5.1
(was: 5.x)
(was: 5.0-rc)
Since Version: NA
Source Control Link:
https://github.com/apache/cassandra/commit/389479c9a22baf8914d4ba63896ce464e7e9ef62
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
That's correct.
Thanks for the review, committed with updated comment.
> Commitlog with direct io fails test_change_durable_writes
> -
>
> Key: CASSANDRA-19471
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19471
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Commit Log
>Reporter: Brandon Williams
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 5.0-beta2, 5.1
>
>
> With the commitlog_disk_access_mode set to direct, and the improved
> configuration_test.py::TestConfiguration::test_change_durable_writes from
> CASSANDRA-19465, this fails with either:
> {noformat}
> AssertionError: Commitlog was written with durable writes disabled
> {noformat}
> Or what appears to be the original exception reported in CASSANDRA-19465:
> {noformat}
> node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,465
> StorageService.java:631 - Stopping native transport
> node1: ERROR [MutationStage-5] 2024-03-14 17:16:08,465
> StorageProxy.java:1670 - Failed to apply mutation locally :
> java.lang.IllegalArgumentException: newPosition > limit: (1048634 > 1048576)
> at java.base/java.nio.Buffer.createPositionException(Buffer.java:341)
> at java.base/java.nio.Buffer.position(Buffer.java:316)
> at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516)
> at
> java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:321)
> at
> java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:73)
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.allocate(CommitLogSegment.java:216)
> at
> org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard.allocate(CommitLogSegmentManagerStandard.java:52)
> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
> at
> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99)
> at
> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53)
> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:612)
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:497)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:244)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:264)
> at
> org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1664)
> at
> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2624)
> at
> org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:833)
> node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,470
> StorageService.java:636 - Stopping gossiper
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch trunk updated (63c6261856 -> 52502002ba)
This is an automated email from the ASF dual-hosted git repository. brandonwilliams pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git from 63c6261856 Reimplement ClusterMetadata::writePlacementAllSettled to step through InProgressSequences to determine state when finished. new 389479c9a2 Align buffer with commitlog segment size new 52502002ba Merge branch 'cassandra-5.0' into trunk The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt | 1 + .../cassandra/db/commitlog/DirectIOSegment.java | 21 - 2 files changed, 13 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch cassandra-5.0 updated: Align buffer with commitlog segment size
This is an automated email from the ASF dual-hosted git repository.
brandonwilliams pushed a commit to branch cassandra-5.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cassandra-5.0 by this push:
new 389479c9a2 Align buffer with commitlog segment size
389479c9a2 is described below
commit 389479c9a22baf8914d4ba63896ce464e7e9ef62
Author: Brandon Williams
AuthorDate: Wed Mar 20 11:21:05 2024 -0500
Align buffer with commitlog segment size
Patch by brandonwilliams and blambov; reviewed by bereng for
CASSANDRA-19471
---
CHANGES.txt | 1 +
.../cassandra/db/commitlog/DirectIOSegment.java | 21 -
2 files changed, 13 insertions(+), 9 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index c056c2785f..0954529c82 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
5.0-beta2
+ * Align buffer with commitlog segment size (CASSANDRA-19471)
* Ensure SAI indexes empty byte buffers for types that allow them as a valid
input (CASSANDRA-19461)
* Deprecate Python 3.7 and earlier, but allow cqlsh to run with Python
3.6-3.11 (CASSANDRA-19467)
* Set uuid_sstable_identifiers_enabled to true in cassandra-latest.yaml
(CASSANDRA-19460)
diff --git a/src/java/org/apache/cassandra/db/commitlog/DirectIOSegment.java
b/src/java/org/apache/cassandra/db/commitlog/DirectIOSegment.java
index 2f350f71fb..b4819aa201 100644
--- a/src/java/org/apache/cassandra/db/commitlog/DirectIOSegment.java
+++ b/src/java/org/apache/cassandra/db/commitlog/DirectIOSegment.java
@@ -184,11 +184,16 @@ public class DirectIOSegment extends CommitLogSegment
@Override
public SimpleCachedBufferPool createBufferPool()
{
-// The direct buffer must be aligned with the file system block
size. We cannot enforce that during
-// allocation, but we can get an aligned slice from the allocated
buffer. The buffer must be oversized by the
-// alignment unit to make it possible.
+// Since the allocated buffer may be offset by up to fsBlockSize -
+// 1 bytes, the buffer must be oversized by that amount. Alignment
+// will make sure both the start and end of the buffer are aligned,
+// cutting off any remaining bytes from both sides. Note that if we
+// oversize by fsBlockSize bytes and the buffer happens to be
+// already aligned, neither the start nor the end of the buffer
+// will be adjusted and the resulting buffer will be bigger than
+// expected.
return new
SimpleCachedBufferPool(DatabaseDescriptor.getCommitLogMaxCompressionBuffersInPool(),
-
DatabaseDescriptor.getCommitLogSegmentSize() + fsBlockSize,
+
DatabaseDescriptor.getCommitLogSegmentSize() + fsBlockSize - 1,
BufferType.OFF_HEAP) {
@Override
public ByteBuffer createBuffer()
@@ -202,12 +207,10 @@ public class DirectIOSegment extends CommitLogSegment
ByteBufferUtil.writeZeroes(original.duplicate(),
original.limit());
ByteBuffer alignedBuffer;
-if (original.alignmentOffset(0, fsBlockSize) > 0)
-alignedBuffer = original.alignedSlice(fsBlockSize);
-else
-alignedBuffer = original.slice().limit(segmentSize);
+alignedBuffer = original.alignedSlice(fsBlockSize);
-assert alignedBuffer.limit() >= segmentSize :
String.format("Bytebuffer slicing failed to get required buffer size
(required=%d, current size=%d", segmentSize, alignedBuffer.limit());
+assert alignedBuffer.limit() == segmentSize :
String.format("Bytebuffer slicing failed to get required buffer size
(required=%d, current size=%d", segmentSize, alignedBuffer.limit());
+assert alignedBuffer.limit() == alignedBuffer.capacity() :
String.format("Bytebuffer limit and capacity do not match (limit=%d,
capacity=%d", alignedBuffer.limit(), alignedBuffer.capacity());
assert alignedBuffer.alignmentOffset(0, fsBlockSize) == 0
: String.format("Index 0 should be aligned to %d page size.", fsBlockSize);
assert
alignedBuffer.alignmentOffset(alignedBuffer.limit(), fsBlockSize) == 0 :
String.format("Limit should be aligned to %d page size", fsBlockSize);
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) 01/01: Merge branch 'cassandra-5.0' into trunk
This is an automated email from the ASF dual-hosted git repository. brandonwilliams pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 52502002ba1340adee8b28bfe5b4d7959a0bca86 Merge: 63c6261856 389479c9a2 Author: Brandon Williams AuthorDate: Mon Mar 25 06:34:13 2024 -0500 Merge branch 'cassandra-5.0' into trunk CHANGES.txt | 1 + .../cassandra/db/commitlog/DirectIOSegment.java | 21 - 2 files changed, 13 insertions(+), 9 deletions(-) diff --cc CHANGES.txt index 1e34cbe71e,0954529c82..e92cd81496 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,30 -1,5 +1,31 @@@ -5.0-beta2 +5.1 + * Fix StorageService::constructRangeToEndpointMap for non-distributed keyspaces (CASSANDRA-19255) + * Group nodetool cms commands into single command group (CASSANDRA-19393) + * Register the measurements of the bootstrap process as Dropwizard metrics (CASSANDRA-19447) + * Add LIST SUPERUSERS CQL statement (CASSANDRA-19417) + * Modernize CQLSH datetime conversions (CASSANDRA-18879) + * Harry model and in-JVM tests for partition-restricted 2i queries (CASSANDRA-18275) + * Refactor cqlshmain global constants (CASSANDRA-19201) + * Remove native_transport_port_ssl (CASSANDRA-19397) + * Make nodetool reconfigurecms sync by default and add --cancel to be able to cancel ongoing reconfigurations (CASSANDRA-19216) + * Expose auth mode in system_views.clients, nodetool clientstats, metrics (CASSANDRA-19366) + * Remove sealed_periods and last_sealed_period tables (CASSANDRA-19189) + * Improve setup and initialisation of LocalLog/LogSpec (CASSANDRA-19271) + * Refactor structure of caching metrics and expose auth cache metrics via JMX (CASSANDRA-17062) + * Allow CQL client certificate authentication to work without sending an AUTHENTICATE request (CASSANDRA-18857) + * Extend nodetool tpstats and system_views.thread_pools with detailed pool parameters (CASSANDRA-19289) + * Remove dependency on Sigar in favor of OSHI (CASSANDRA-16565) + * Simplify the bind marker and Term logic (CASSANDRA-18813) + * Limit cassandra startup to supported JDKs, allow higher JDKs by setting CASSANDRA_JDK_UNSUPPORTED (CASSANDRA-18688) + * Standardize nodetool tablestats formatting of data units (CASSANDRA-19104) + * Make nodetool tablestats use number of significant digits for time and average values consistently (CASSANDRA-19015) + * Upgrade jackson to 2.15.3 and snakeyaml to 2.1 (CASSANDRA-18875) + * Transactional Cluster Metadata [CEP-21] (CASSANDRA-18330) + * Add ELAPSED command to cqlsh (CASSANDRA-18861) + * Add the ability to disable bulk loading of SSTables (CASSANDRA-18781) + * Clean up obsolete functions and simplify cql_version handling in cqlsh (CASSANDRA-18787) +Merged from 5.0: + * Align buffer with commitlog segment size (CASSANDRA-19471) * Ensure SAI indexes empty byte buffers for types that allow them as a valid input (CASSANDRA-19461) * Deprecate Python 3.7 and earlier, but allow cqlsh to run with Python 3.6-3.11 (CASSANDRA-19467) * Set uuid_sstable_identifiers_enabled to true in cassandra-latest.yaml (CASSANDRA-19460) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra-dtest) branch trunk updated: Durable writes begins measuring after schema
This is an automated email from the ASF dual-hosted git repository.
brandonwilliams pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra-dtest.git
The following commit(s) were added to refs/heads/trunk by this push:
new ff2a673b Durable writes begins measuring after schema
ff2a673b is described below
commit ff2a673b37cfe1af0fe97b09cfe4a348ca536d08
Author: Brandon Williams
AuthorDate: Fri Mar 22 10:31:15 2024 -0500
Durable writes begins measuring after schema
Patch by brandonwilliams; reviewed by bereng for CASSANDRA-19471
---
configuration_test.py | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/configuration_test.py b/configuration_test.py
index d307d2a7..277be607 100644
--- a/configuration_test.py
+++ b/configuration_test.py
@@ -89,15 +89,17 @@ class TestConfiguration(Tester):
cluster.stop()
cluster.clear()
+cluster.set_batch_commitlog(enabled=True, use_batch_window =
cluster.version() < '5.0')
+
cluster.set_configuration_options(values={'commitlog_segment_size_in_mb': 1})
cluster.start()
node = cluster.nodelist()[0]
-init_size = commitlog_size(node)
session = self.patient_exclusive_cql_connection(node)
# set up a keyspace without durable writes, then alter it to use them
session.execute("CREATE KEYSPACE ks WITH REPLICATION = {'class':
'SimpleStrategy', 'replication_factor': 1} "
"AND DURABLE_WRITES = false")
session.execute('CREATE TABLE ks.tab (key int PRIMARY KEY, a int, b
int, c int)')
+init_size = commitlog_size(node)
write_to_trigger_fsync(session, 'ks', 'tab')
assert commitlog_size(node) == init_size, "Commitlog was written with
durable writes disabled"
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19471) Commitlog with direct io fails test_change_durable_writes
[
https://issues.apache.org/jira/browse/CASSANDRA-19471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830431#comment-17830431
]
Berenguer Blasi commented on CASSANDRA-19471:
-
If so +1
> Commitlog with direct io fails test_change_durable_writes
> -
>
> Key: CASSANDRA-19471
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19471
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Commit Log
>Reporter: Brandon Williams
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 5.0-rc, 5.x
>
>
> With the commitlog_disk_access_mode set to direct, and the improved
> configuration_test.py::TestConfiguration::test_change_durable_writes from
> CASSANDRA-19465, this fails with either:
> {noformat}
> AssertionError: Commitlog was written with durable writes disabled
> {noformat}
> Or what appears to be the original exception reported in CASSANDRA-19465:
> {noformat}
> node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,465
> StorageService.java:631 - Stopping native transport
> node1: ERROR [MutationStage-5] 2024-03-14 17:16:08,465
> StorageProxy.java:1670 - Failed to apply mutation locally :
> java.lang.IllegalArgumentException: newPosition > limit: (1048634 > 1048576)
> at java.base/java.nio.Buffer.createPositionException(Buffer.java:341)
> at java.base/java.nio.Buffer.position(Buffer.java:316)
> at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516)
> at
> java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:321)
> at
> java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:73)
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.allocate(CommitLogSegment.java:216)
> at
> org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard.allocate(CommitLogSegmentManagerStandard.java:52)
> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
> at
> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99)
> at
> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53)
> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:612)
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:497)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:244)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:264)
> at
> org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1664)
> at
> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2624)
> at
> org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:833)
> node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,470
> StorageService.java:636 - Stopping gossiper
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19343) Test Failure: org.apache.cassandra.fuzz.ring.ConsistentBootstrapTest.coordinatorIsBehindTest
[
https://issues.apache.org/jira/browse/CASSANDRA-19343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830430#comment-17830430
]
Alex Petrov commented on CASSANDRA-19343:
-
+1, thank you for root-causing!
> Test Failure:
> org.apache.cassandra.fuzz.ring.ConsistentBootstrapTest.coordinatorIsBehindTest
>
>
> Key: CASSANDRA-19343
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19343
> Project: Cassandra
> Issue Type: Bug
> Components: CI
>Reporter: Ekaterina Dimitrova
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 5.x
>
> Attachments: ci_summary.html, result_details.tar.gz
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java}
> java.lang.IllegalStateException: Can't use shutdown instances, delegate is
> null at
> org.apache.cassandra.distributed.impl.AbstractCluster$Wrapper.delegate(AbstractCluster.java:283)
> at
> org.apache.cassandra.distributed.impl.DelegatingInvokableInstance.transfer(DelegatingInvokableInstance.java:49)
> at
> org.apache.cassandra.distributed.api.IInvokableInstance.runsOnInstance(IInvokableInstance.java:45)
> at
> org.apache.cassandra.distributed.api.IInvokableInstance.runOnInstance(IInvokableInstance.java:46)
> at
> org.apache.cassandra.distributed.shared.ClusterUtils.unpauseCommits(ClusterUtils.java:548)
> at
> org.apache.cassandra.fuzz.ring.ConsistentBootstrapTest.coordinatorIsBehindTest(ConsistentBootstrapTest.java:227)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
> https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2636/workflows/93adbf3e-acf8-4a62-a1f6-baf4f4689347/jobs/54912/tests
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19477) Do not go to disk to get HintsStore.getTotalFileSize
[
https://issues.apache.org/jira/browse/CASSANDRA-19477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Yeschenko updated CASSANDRA-19477:
--
Status: Ready to Commit (was: Review In Progress)
+1, good stuff
> Do not go to disk to get HintsStore.getTotalFileSize
>
>
> Key: CASSANDRA-19477
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19477
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Hints
>Reporter: Jon Haddad
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.1.x, 5.0-rc, 5.x
>
> Attachments: flame-cassandra0-patched-2024-03-25_00-40-47.html,
> flame-cassandra0-release-2024-03-25_00-16-44.html, flamegraph.cpu.html,
> image-2024-03-24-17-57-32-560.png, image-2024-03-24-18-08-36-918.png,
> image-2024-03-24-18-16-50-370.png, image-2024-03-24-18-17-48-334.png,
> image-2024-03-24-18-20-07-734.png
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> When testing a cluster with more requests than it could handle, I noticed
> significant CPU time (25%) spent in HintsStore.getTotalFileSize. Here's what
> I'm seeing from profiling:
> 10% of CPU time spent in HintsDescriptor.fileName which only does this:
>
> {noformat}
> return String.format("%s-%s-%s.hints", hostId, timestamp, version);{noformat}
> At a bare minimum here we should create this string up front with the host
> and version and eliminate 2 of the 3 substitutions, but I think it's probably
> faster to use a StringBuilder and avoid the underlying regular expression
> altogether.
> 12% of the time is spent in org.apache.cassandra.io.util.File.length. It
> looks like this is called once for each hint file on disk for each host we're
> hinting to. In the case of an overloaded cluster, this is significant. It
> would be better if we were to track the file size in memory for each hint
> file and reference that rather than go to the filesystem.
> These fairly small changes should make Cassandra more reliable when under
> load spikes.
> CPU Flame graph attached.
> I only tested this in 4.1 but it looks like this is present up to trunk.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19477) Do not go to disk to get HintsStore.getTotalFileSize
[
https://issues.apache.org/jira/browse/CASSANDRA-19477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Yeschenko updated CASSANDRA-19477:
--
Status: Patch Available (was: Needs Committer)
> Do not go to disk to get HintsStore.getTotalFileSize
>
>
> Key: CASSANDRA-19477
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19477
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Hints
>Reporter: Jon Haddad
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.1.x, 5.0-rc, 5.x
>
> Attachments: flame-cassandra0-patched-2024-03-25_00-40-47.html,
> flame-cassandra0-release-2024-03-25_00-16-44.html, flamegraph.cpu.html,
> image-2024-03-24-17-57-32-560.png, image-2024-03-24-18-08-36-918.png,
> image-2024-03-24-18-16-50-370.png, image-2024-03-24-18-17-48-334.png,
> image-2024-03-24-18-20-07-734.png
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> When testing a cluster with more requests than it could handle, I noticed
> significant CPU time (25%) spent in HintsStore.getTotalFileSize. Here's what
> I'm seeing from profiling:
> 10% of CPU time spent in HintsDescriptor.fileName which only does this:
>
> {noformat}
> return String.format("%s-%s-%s.hints", hostId, timestamp, version);{noformat}
> At a bare minimum here we should create this string up front with the host
> and version and eliminate 2 of the 3 substitutions, but I think it's probably
> faster to use a StringBuilder and avoid the underlying regular expression
> altogether.
> 12% of the time is spent in org.apache.cassandra.io.util.File.length. It
> looks like this is called once for each hint file on disk for each host we're
> hinting to. In the case of an overloaded cluster, this is significant. It
> would be better if we were to track the file size in memory for each hint
> file and reference that rather than go to the filesystem.
> These fairly small changes should make Cassandra more reliable when under
> load spikes.
> CPU Flame graph attached.
> I only tested this in 4.1 but it looks like this is present up to trunk.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19477) Do not go to disk to get HintsStore.getTotalFileSize
[
https://issues.apache.org/jira/browse/CASSANDRA-19477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Yeschenko updated CASSANDRA-19477:
--
Reviewers: Aleksey Yeschenko, Aleksey Yeschenko (was: Aleksey Yeschenko)
Aleksey Yeschenko, Aleksey Yeschenko (was: Aleksey Yeschenko)
Status: Review In Progress (was: Patch Available)
> Do not go to disk to get HintsStore.getTotalFileSize
>
>
> Key: CASSANDRA-19477
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19477
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Hints
>Reporter: Jon Haddad
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.1.x, 5.0-rc, 5.x
>
> Attachments: flame-cassandra0-patched-2024-03-25_00-40-47.html,
> flame-cassandra0-release-2024-03-25_00-16-44.html, flamegraph.cpu.html,
> image-2024-03-24-17-57-32-560.png, image-2024-03-24-18-08-36-918.png,
> image-2024-03-24-18-16-50-370.png, image-2024-03-24-18-17-48-334.png,
> image-2024-03-24-18-20-07-734.png
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> When testing a cluster with more requests than it could handle, I noticed
> significant CPU time (25%) spent in HintsStore.getTotalFileSize. Here's what
> I'm seeing from profiling:
> 10% of CPU time spent in HintsDescriptor.fileName which only does this:
>
> {noformat}
> return String.format("%s-%s-%s.hints", hostId, timestamp, version);{noformat}
> At a bare minimum here we should create this string up front with the host
> and version and eliminate 2 of the 3 substitutions, but I think it's probably
> faster to use a StringBuilder and avoid the underlying regular expression
> altogether.
> 12% of the time is spent in org.apache.cassandra.io.util.File.length. It
> looks like this is called once for each hint file on disk for each host we're
> hinting to. In the case of an overloaded cluster, this is significant. It
> would be better if we were to track the file size in memory for each hint
> file and reference that rather than go to the filesystem.
> These fairly small changes should make Cassandra more reliable when under
> load spikes.
> CPU Flame graph attached.
> I only tested this in 4.1 but it looks like this is present up to trunk.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19193) Reimplement ClusterMetadata::writePlacementAllSettled
[ https://issues.apache.org/jira/browse/CASSANDRA-19193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-19193: Status: Ready to Commit (was: Review In Progress) > Reimplement ClusterMetadata::writePlacementAllSettled > - > > Key: CASSANDRA-19193 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19193 > Project: Cassandra > Issue Type: Improvement > Components: Transactional Cluster Metadata >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 5.1-alpha1 > > Attachments: ci_summary.html, result_details.tar.gz > > Time Spent: 10m > Remaining Estimate: 0h > > This should step through InProgressSequences to determine state when > finished, rather than emulating the logic inline. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19193) Reimplement ClusterMetadata::writePlacementAllSettled
[ https://issues.apache.org/jira/browse/CASSANDRA-19193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-19193: Reviewers: Alex Petrov (was: Alex Petrov, Sam Tunnicliffe) Status: Review In Progress (was: Patch Available) > Reimplement ClusterMetadata::writePlacementAllSettled > - > > Key: CASSANDRA-19193 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19193 > Project: Cassandra > Issue Type: Improvement > Components: Transactional Cluster Metadata >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 5.1-alpha1 > > Attachments: ci_summary.html, result_details.tar.gz > > Time Spent: 10m > Remaining Estimate: 0h > > This should step through InProgressSequences to determine state when > finished, rather than emulating the logic inline. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19193) Reimplement ClusterMetadata::writePlacementAllSettled
[ https://issues.apache.org/jira/browse/CASSANDRA-19193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-19193: Source Control Link: https://github.com/apache/cassandra/commit/63c62618560ad65b5b3e9f4d34b70b8b6dd0a75b Resolution: Fixed Status: Resolved (was: Ready to Commit) and committed, thanks > Reimplement ClusterMetadata::writePlacementAllSettled > - > > Key: CASSANDRA-19193 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19193 > Project: Cassandra > Issue Type: Improvement > Components: Transactional Cluster Metadata >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 5.1-alpha1 > > Attachments: ci_summary.html, result_details.tar.gz > > Time Spent: 10m > Remaining Estimate: 0h > > This should step through InProgressSequences to determine state when > finished, rather than emulating the logic inline. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch trunk updated: Reimplement ClusterMetadata::writePlacementAllSettled to step through InProgressSequences to determine state when finished.
This is an automated email from the ASF dual-hosted git repository.
marcuse pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new 63c6261856 Reimplement ClusterMetadata::writePlacementAllSettled to
step through InProgressSequences to determine state when finished.
63c6261856 is described below
commit 63c62618560ad65b5b3e9f4d34b70b8b6dd0a75b
Author: Marcus Eriksson
AuthorDate: Tue Mar 12 08:31:05 2024 +0100
Reimplement ClusterMetadata::writePlacementAllSettled to step through
InProgressSequences to determine state when finished.
Patch by marcuse; reviewed by Alex Petrov for CASSANDRA-19193
---
.../org/apache/cassandra/tcm/ClusterMetadata.java | 46 +++--
.../apache/cassandra/tcm/MultiStepOperation.java | 34 +
.../apache/cassandra/tcm/sequences/AddToCMS.java | 6 +++
.../cassandra/tcm/sequences/BootstrapAndJoin.java | 7 +++
.../tcm/sequences/BootstrapAndReplace.java | 7 +++
.../tcm/sequences/InProgressSequences.java | 13 +++--
.../org/apache/cassandra/tcm/sequences/Move.java | 10 +++-
.../cassandra/tcm/sequences/ReconfigureCMS.java| 21
.../tcm/sequences/UnbootstrapAndLeave.java | 7 +++
.../cassandra/tcm/transformations/UnsafeJoin.java | 32 ++--
.../test/log/MetadataChangeSimulationTest.java | 58 +-
.../apache/cassandra/tcm/ClusterMetadataTest.java | 2 +-
12 files changed, 170 insertions(+), 73 deletions(-)
diff --git a/src/java/org/apache/cassandra/tcm/ClusterMetadata.java
b/src/java/org/apache/cassandra/tcm/ClusterMetadata.java
index 5e6b9a6060..3224b29466 100644
--- a/src/java/org/apache/cassandra/tcm/ClusterMetadata.java
+++ b/src/java/org/apache/cassandra/tcm/ClusterMetadata.java
@@ -23,6 +23,7 @@ import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
import java.util.HashSet;
+import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Objects;
@@ -64,7 +65,6 @@ import
org.apache.cassandra.tcm.ownership.PrimaryRangeComparator;
import org.apache.cassandra.tcm.ownership.PlacementForRange;
import org.apache.cassandra.tcm.ownership.TokenMap;
import org.apache.cassandra.tcm.ownership.VersionedEndpoints;
-import org.apache.cassandra.tcm.sequences.BootstrapAndJoin;
import org.apache.cassandra.tcm.sequences.InProgressSequences;
import org.apache.cassandra.tcm.sequences.LockedRanges;
import org.apache.cassandra.tcm.serialization.MetadataSerializer;
@@ -309,45 +309,15 @@ public class ClusterMetadata
public DataPlacement writePlacementAllSettled(KeyspaceMetadata ksm)
{
-List joining = new ArrayList<>();
-List leaving = new ArrayList<>();
-List moving = new ArrayList<>();
-
-for (Map.Entry entry : directory.states.entrySet())
-{
-switch (entry.getValue())
-{
-case BOOTSTRAPPING:
-joining.add(entry.getKey());
-break;
-case LEAVING:
-leaving.add(entry.getKey());
-break;
-case MOVING:
-moving.add(entry.getKey());
-break;
-}
-}
-
-Transformer t = transformer();
-for (NodeId node: joining)
+ClusterMetadata metadata = this;
+Iterator> iter =
metadata.inProgressSequences.iterator();
+while (iter.hasNext())
{
-MultiStepOperation joinSequence = inProgressSequences.get(node);
-assert joinSequence instanceof BootstrapAndJoin;
-Set tokens =
((BootstrapAndJoin)joinSequence).finishJoin.tokens;
-t = t.proposeToken(node, tokens);
+Transformation.Result result = iter.next().applyTo(metadata);
+assert result.isSuccess();
+metadata = result.success().metadata;
}
-for (NodeId node : leaving)
-t = t.proposeRemoveNode(node);
-// todo: add tests for move!
-for (NodeId node : moving)
-t = t.proposeRemoveNode(node).proposeToken(node,
tokenMap.tokens(node));
-
-ClusterMetadata proposed = t.build().metadata;
-return ClusterMetadataService.instance()
- .placementProvider()
- .calculatePlacements(epoch,
proposed.tokenMap.toRanges(), proposed, Keyspaces.of(ksm))
- .get(ksm.params.replication);
+return metadata.placements.get(ksm.params.replication);
}
// TODO Remove this as it isn't really an equivalent to the previous
concept of pending ranges
diff --git a/src/java/org/apache/cassandra/tcm/MultiStepOperation.java
b/src/java/org/apache/cassandra/tcm/MultiStepOperation.java
index b315aa9b42..9e8f21c6ec 100644
--- a/src/java/org
[jira] [Comment Edited] (CASSANDRA-19193) Reimplement ClusterMetadata::writePlacementAllSettled
[
https://issues.apache.org/jira/browse/CASSANDRA-19193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830416#comment-17830416
]
Alex Petrov edited comment on CASSANDRA-19193 at 3/25/24 9:50 AM:
--
+1, new test looks great!
very small nit: for consistency, maybe rename {{movingNode}} to {{toMove}} and
{{joiningNode}} to {{toJoin}}? Or the other two ({{toReplace}} and {{toLeave}})
to {{replacedNode}} and {{leavingNode}}. Either way.
was (Author: ifesdjeen):
+1, new test looks great!
> Reimplement ClusterMetadata::writePlacementAllSettled
> -
>
> Key: CASSANDRA-19193
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19193
> Project: Cassandra
> Issue Type: Improvement
> Components: Transactional Cluster Metadata
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 5.1-alpha1
>
> Attachments: ci_summary.html, result_details.tar.gz
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This should step through InProgressSequences to determine state when
> finished, rather than emulating the logic inline.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19193) Reimplement ClusterMetadata::writePlacementAllSettled
[ https://issues.apache.org/jira/browse/CASSANDRA-19193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830416#comment-17830416 ] Alex Petrov commented on CASSANDRA-19193: - +1, new test looks great! > Reimplement ClusterMetadata::writePlacementAllSettled > - > > Key: CASSANDRA-19193 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19193 > Project: Cassandra > Issue Type: Improvement > Components: Transactional Cluster Metadata >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 5.1-alpha1 > > Attachments: ci_summary.html, result_details.tar.gz > > Time Spent: 10m > Remaining Estimate: 0h > > This should step through InProgressSequences to determine state when > finished, rather than emulating the logic inline. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19477) Do not go to disk to get HintsStore.getTotalFileSize
[
https://issues.apache.org/jira/browse/CASSANDRA-19477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic updated CASSANDRA-19477:
--
Status: Needs Committer (was: Patch Available)
> Do not go to disk to get HintsStore.getTotalFileSize
>
>
> Key: CASSANDRA-19477
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19477
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Hints
>Reporter: Jon Haddad
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.1.x, 5.0-rc, 5.x
>
> Attachments: flame-cassandra0-patched-2024-03-25_00-40-47.html,
> flame-cassandra0-release-2024-03-25_00-16-44.html, flamegraph.cpu.html,
> image-2024-03-24-17-57-32-560.png, image-2024-03-24-18-08-36-918.png,
> image-2024-03-24-18-16-50-370.png, image-2024-03-24-18-17-48-334.png,
> image-2024-03-24-18-20-07-734.png
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> When testing a cluster with more requests than it could handle, I noticed
> significant CPU time (25%) spent in HintsStore.getTotalFileSize. Here's what
> I'm seeing from profiling:
> 10% of CPU time spent in HintsDescriptor.fileName which only does this:
>
> {noformat}
> return String.format("%s-%s-%s.hints", hostId, timestamp, version);{noformat}
> At a bare minimum here we should create this string up front with the host
> and version and eliminate 2 of the 3 substitutions, but I think it's probably
> faster to use a StringBuilder and avoid the underlying regular expression
> altogether.
> 12% of the time is spent in org.apache.cassandra.io.util.File.length. It
> looks like this is called once for each hint file on disk for each host we're
> hinting to. In the case of an overloaded cluster, this is significant. It
> would be better if we were to track the file size in memory for each hint
> file and reference that rather than go to the filesystem.
> These fairly small changes should make Cassandra more reliable when under
> load spikes.
> CPU Flame graph attached.
> I only tested this in 4.1 but it looks like this is present up to trunk.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19477) Do not go to disk to get HintsStore.getTotalFileSize
[
https://issues.apache.org/jira/browse/CASSANDRA-19477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830401#comment-17830401
]
Stefan Miklosovic commented on CASSANDRA-19477:
---
[CASSANDRA-19477-trunk|https://github.com/instaclustr/cassandra/tree/CASSANDRA-19477-trunk]
{noformat}
java17_pre-commit_tests
✓ j17_build3m 57s
✓ j17_cqlsh_dtests_py3117m 2s
✓ j17_cqlsh_dtests_py311_vnode 7m 32s
✓ j17_cqlsh_dtests_py386m 50s
✓ j17_cqlsh_dtests_py38_vnode 7m 16s
✓ j17_cqlshlib_cython_tests7m 39s
✓ j17_cqlshlib_tests 6m 31s
✓ j17_dtests 34m 33s
✓ j17_dtests_vnode35m 10s
✓ j17_jvm_dtests_latest_vnode_repeat 26m 31s
✓ j17_jvm_dtests_repeat28m 7s
✓ j17_unit_tests 16m 26s
✓ j17_unit_tests_repeat0m 18s
✓ j17_utests_latest 13m 59s
✓ j17_utests_latest_repeat 0m 13s
✓ j17_utests_oa_repeat 0m 29s
✕ j17_dtests_latest 34m 36s
offline_tools_test.TestOfflineTools test_sstablelevelreset
offline_tools_test.TestOfflineTools test_sstableofflinerelevel
configuration_test.TestConfiguration test_change_durable_writes
configuration_test.TestConfiguration test_change_durable_writes
✕ j17_jvm_dtests 27m 59s
org.apache.cassandra.distributed.test.NativeTransportEncryptionOptionsTest
testEndpointVerificationEnabledIpNotInSAN TIMEOUTED
✕ j17_jvm_dtests_latest_vnode 22m 44s
junit.framework.TestSuite
org.apache.cassandra.fuzz.harry.integration.model.InJVMTokenAwareExecutorTest
TIMEOUTED
✕ j17_utests_oa 13m 58s
org.apache.cassandra.db.compaction.CompactionsBytemanTest
testSSTableNotEnoughDiskSpaceForCompactionGetsDropped
java17_separate_tests
java11_pre-commit_tests
✓ j11_build7m 57s
✓ j11_cqlsh_dtests_py3117m 7s
✓ j11_cqlsh_dtests_py311_vnode10m 13s
✓ j11_cqlsh_dtests_py38 8m 1s
✓ j11_cqlsh_dtests_py38_vnode 10m 25s
✓ j11_cqlshlib_cython_tests7m 28s
✓ j11_cqlshlib_tests 9m 40s
✓ j11_dtests_vnode36m 58s
✓ j11_jvm_dtests_latest_vnode 25m 28s
✓ j11_jvm_dtests_latest_vnode_repeat 29m 22s
✓ j11_jvm_dtests_repeat28m 7s
✓ j11_unit_tests 15m 17s
✓ j11_unit_tests_repeat0m 30s
✓ j11_utests_latest 16m 56s
✓ j11_utests_latest_repeat 0m 34s
✓ j11_utests_oa 13m 58s
✓ j11_utests_oa_repeat 1m 0s
✓ j11_utests_system_keyspace_directory 18m 1s
✓ j11_utests_system_keyspace_directory_repeat 3m 39s
✓ j17_cqlsh_dtests_py3117m 6s
✓ j17_cqlsh_dtests_py311_vnode 7m 27s
✓ j17_cqlsh_dtests_py386m 51s
✓ j17_cqlsh_dtests_py38_vnode 7m 14s
✓ j17_cqlshlib_cython_tests7m 38s
✓ j17_cqlshlib_tests 6m 57s
✓ j17_dtests 32m 21s
✓ j17_dtests_vnode34m 24s
✓ j17_jvm_dtests_latest_vnode 22m 45s
✓ j17_jvm_dtests_latest_vnode_repeat 26m 32s
✓ j17_jvm_dtests_repeat 28m 21s
✓ j17_unit_tests_repeat0m 16s
✓ j17_utests_latest 15m 34s
✓ j17_utests_latest_repeat 0m 36s
✓ j17_utests_oa 13m 43s
✓ j17_utests_oa_repeat 0m 17s
✕ j11_dtests 37m 26s
pushed_notifications_test.TestPushedNotifications
test_move_single_node_localhost
✕ j11_dtests_latest 40m 40s
bootstrap_test.TestBootstrap test_bootstrap_with_reset_bootstrap_state
offline_tools_test.TestOfflineTools test_sstablelevelreset
offline_tools_test.TestOfflineTools test_sstableofflinerelevel
configuration_test.TestConfiguration test_change_durable_writes
✕ j11_jvm_dtests
[jira] [Updated] (CASSANDRA-19153) Preclude irrecoverable log corruption in case split-brain situation during leader election with absent seeds
[ https://issues.apache.org/jira/browse/CASSANDRA-19153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-19153: Status: Review In Progress (was: Patch Available) > Preclude irrecoverable log corruption in case split-brain situation during > leader election with absent seeds > > > Key: CASSANDRA-19153 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19153 > Project: Cassandra > Issue Type: Improvement > Components: Transactional Cluster Metadata >Reporter: Alex Petrov >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 5.x > > Attachments: ci_summary.html, result_details.tar.gz > > Time Spent: 20m > Remaining Estimate: 0h > > It should be possible to detect a scenario where two partitioned nodes > independently elect themselves as the first CMS nodes in a brand new cluster. > In such a case, metadata changes should not be applied so that the conflict > can be resolved. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19153) Preclude irrecoverable log corruption in case split-brain situation during leader election with absent seeds
[ https://issues.apache.org/jira/browse/CASSANDRA-19153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-19153: Source Control Link: https://github.com/apache/cassandra/commit/0ec5ef2c7035fc93323816140994617a9d953956 Resolution: Fixed Status: Resolved (was: Ready to Commit) and committed > Preclude irrecoverable log corruption in case split-brain situation during > leader election with absent seeds > > > Key: CASSANDRA-19153 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19153 > Project: Cassandra > Issue Type: Improvement > Components: Transactional Cluster Metadata >Reporter: Alex Petrov >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 5.x > > Attachments: ci_summary.html, result_details.tar.gz > > Time Spent: 20m > Remaining Estimate: 0h > > It should be possible to detect a scenario where two partitioned nodes > independently elect themselves as the first CMS nodes in a brand new cluster. > In such a case, metadata changes should not be applied so that the conflict > can be resolved. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19153) Preclude irrecoverable log corruption in case split-brain situation during leader election with absent seeds
[ https://issues.apache.org/jira/browse/CASSANDRA-19153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-19153: Status: Ready to Commit (was: Review In Progress) > Preclude irrecoverable log corruption in case split-brain situation during > leader election with absent seeds > > > Key: CASSANDRA-19153 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19153 > Project: Cassandra > Issue Type: Improvement > Components: Transactional Cluster Metadata >Reporter: Alex Petrov >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 5.x > > Attachments: ci_summary.html, result_details.tar.gz > > Time Spent: 20m > Remaining Estimate: 0h > > It should be possible to detect a scenario where two partitioned nodes > independently elect themselves as the first CMS nodes in a brand new cluster. > In such a case, metadata changes should not be applied so that the conflict > can be resolved. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch trunk updated: Preclude irrecoverable log corruption in case split-brain situation during leader election with absent seeds
This is an automated email from the ASF dual-hosted git repository.
marcuse pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new 0ec5ef2c70 Preclude irrecoverable log corruption in case split-brain
situation during leader election with absent seeds
0ec5ef2c70 is described below
commit 0ec5ef2c7035fc93323816140994617a9d953956
Author: Alex Petrov
AuthorDate: Wed Mar 6 07:58:18 2024 +0100
Preclude irrecoverable log corruption in case split-brain situation during
leader election with absent seeds
Patch by Alex Petrov; reviewed my marcuse for CASSANDRA-19153
---
src/java/org/apache/cassandra/config/Config.java | 1 +
.../cassandra/config/DatabaseDescriptor.java | 5 +
.../net/CMSIdentifierMismatchException.java| 32 +++
.../cassandra/net/InboundMessageHandler.java | 16
.../org/apache/cassandra/net/MessageDelivery.java | 7 +-
.../org/apache/cassandra/tcm/ClusterMetadata.java | 104 +++--
src/java/org/apache/cassandra/tcm/Commit.java | 19 +++-
src/java/org/apache/cassandra/tcm/Discovery.java | 33 +--
src/java/org/apache/cassandra/tcm/Startup.java | 9 +-
.../org/apache/cassandra/tcm/log/LogState.java | 10 +-
.../tcm/migration/ClusterMetadataHolder.java | 4 +-
.../tcm/serialization/MessageSerializers.java | 5 +
.../cassandra/distributed/impl/InstanceConfig.java | 1 +
.../distributed/test/log/RegisterTest.java | 1 -
.../distributed/test/tcm/SplitBrainTest.java | 79
15 files changed, 300 insertions(+), 26 deletions(-)
diff --git a/src/java/org/apache/cassandra/config/Config.java
b/src/java/org/apache/cassandra/config/Config.java
index 1b2ea89512..75bbdd73c8 100644
--- a/src/java/org/apache/cassandra/config/Config.java
+++ b/src/java/org/apache/cassandra/config/Config.java
@@ -1296,5 +1296,6 @@ public class Config
public volatile DurationSpec.LongMillisecondsBound
progress_barrier_timeout = new DurationSpec.LongMillisecondsBound("360ms");
public volatile DurationSpec.LongMillisecondsBound
progress_barrier_backoff = new DurationSpec.LongMillisecondsBound("1000ms");
+public volatile DurationSpec.LongSecondsBound discovery_timeout = new
DurationSpec.LongSecondsBound("30s");
public boolean unsafe_tcm_mode = false;
}
diff --git a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index 39ddbe5046..59f4040203 100644
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
@@ -5134,6 +5134,11 @@ public class DatabaseDescriptor
conf.progress_barrier_backoff = new
DurationSpec.LongMillisecondsBound(timeOutInMillis);
}
+public static long getDiscoveryTimeout(TimeUnit unit)
+{
+return conf.discovery_timeout.to(unit);
+}
+
public static boolean getUnsafeTCMMode()
{
return conf.unsafe_tcm_mode;
diff --git
a/src/java/org/apache/cassandra/net/CMSIdentifierMismatchException.java
b/src/java/org/apache/cassandra/net/CMSIdentifierMismatchException.java
new file mode 100644
index 00..6f525b1a90
--- /dev/null
+++ b/src/java/org/apache/cassandra/net/CMSIdentifierMismatchException.java
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.net;
+
+/**
+ * Exception thrown in case of a CMS identifier mismatch. This should usually
not happen, except rare cases of
+ * network partition during CMS election during initial cluster bringup. This
is just a precaution to avoid
+ * corrupting CMS log.
+ */
+public class CMSIdentifierMismatchException extends RuntimeException
+{
+public CMSIdentifierMismatchException(String format)
+{
+super(format);
+}
+}
diff --git a/src/java/org/apache/cassandra/net/InboundMessageHandler.java
b/src/java/org/apache/cassandra/net/InboundMessageHandler.java
index ce75b67656..746c0f7a39 100644
--- a/src/java/org/apache/cassandra/net/InboundMessageHandler.java
+++ b/src/ja
[jira] [Commented] (CASSANDRA-19471) Commitlog with direct io fails test_change_durable_writes
[
https://issues.apache.org/jira/browse/CASSANDRA-19471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830395#comment-17830395
]
Berenguer Blasi commented on CASSANDRA-19471:
-
Review is for these correct?
- PR
https://github.com/driftx/cassandra/commit/3659730ac1cfcadada46677b19b5e714531a2984
- dtests PR
https://github.com/driftx/cassandra-dtest/commit/2499a609b7a8bd750417c3fa56855ab1e9e800f1
- CI
https://app.circleci.com/pipelines/github/driftx/cassandra/1536/workflows/150c6753-e905-4503-9f58-f6c7e8357881
> Commitlog with direct io fails test_change_durable_writes
> -
>
> Key: CASSANDRA-19471
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19471
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Commit Log
>Reporter: Brandon Williams
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 5.0-rc, 5.x
>
>
> With the commitlog_disk_access_mode set to direct, and the improved
> configuration_test.py::TestConfiguration::test_change_durable_writes from
> CASSANDRA-19465, this fails with either:
> {noformat}
> AssertionError: Commitlog was written with durable writes disabled
> {noformat}
> Or what appears to be the original exception reported in CASSANDRA-19465:
> {noformat}
> node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,465
> StorageService.java:631 - Stopping native transport
> node1: ERROR [MutationStage-5] 2024-03-14 17:16:08,465
> StorageProxy.java:1670 - Failed to apply mutation locally :
> java.lang.IllegalArgumentException: newPosition > limit: (1048634 > 1048576)
> at java.base/java.nio.Buffer.createPositionException(Buffer.java:341)
> at java.base/java.nio.Buffer.position(Buffer.java:316)
> at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516)
> at
> java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:321)
> at
> java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:73)
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.allocate(CommitLogSegment.java:216)
> at
> org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard.allocate(CommitLogSegmentManagerStandard.java:52)
> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
> at
> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99)
> at
> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53)
> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:612)
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:497)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:244)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:264)
> at
> org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1664)
> at
> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2624)
> at
> org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:833)
> node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,470
> StorageService.java:636 - Stopping gossiper
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
