[jira] [Updated] (CASSANDRASC-86) Startup Validation Failures when Checking Sidecar Connectivity
[ https://issues.apache.org/jira/browse/CASSANDRASC-86?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francisco Guerrero updated CASSANDRASC-86: -- Fix Version/s: 1.0 Source Control Link: https://github.com/apache/cassandra-sidecar/commit/2eb3474d7037a2887bcd9dee1f64c2a36a7e8d26 Resolution: Fixed Status: Resolved (was: Ready to Commit) > Startup Validation Failures when Checking Sidecar Connectivity > -- > > Key: CASSANDRASC-86 > URL: https://issues.apache.org/jira/browse/CASSANDRASC-86 > Project: Sidecar for Apache Cassandra > Issue Type: Improvement > Components: Configuration >Reporter: Yuriy Semchyshyn >Assignee: Yuriy Semchyshyn >Priority: Normal > Labels: pull-request-available > Fix For: 1.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > We have experienced repeated startup validation failures caused by Sidecar > health checks for some jobs with a large number of Spark executors. > These failures are likely caused by the thundering herd problem, and have > been so far worked around by disabling startup validations altogether. > In order to prevent them going forward, a random delay needs to be added > between retries of health checks in Sidecar client. > It is also worth increasing the overall timeout for Sidecar health checks > from current 30 seconds to 60 seconds. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRASC-86) Startup Validation Failures when Checking Sidecar Connectivity
[ https://issues.apache.org/jira/browse/CASSANDRASC-86?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francisco Guerrero updated CASSANDRASC-86: -- Status: Ready to Commit (was: Review In Progress) > Startup Validation Failures when Checking Sidecar Connectivity > -- > > Key: CASSANDRASC-86 > URL: https://issues.apache.org/jira/browse/CASSANDRASC-86 > Project: Sidecar for Apache Cassandra > Issue Type: Improvement > Components: Configuration >Reporter: Yuriy Semchyshyn >Assignee: Yuriy Semchyshyn >Priority: Normal > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > We have experienced repeated startup validation failures caused by Sidecar > health checks for some jobs with a large number of Spark executors. > These failures are likely caused by the thundering herd problem, and have > been so far worked around by disabling startup validations altogether. > In order to prevent them going forward, a random delay needs to be added > between retries of health checks in Sidecar client. > It is also worth increasing the overall timeout for Sidecar health checks > from current 30 seconds to 60 seconds. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRASC-86) Startup Validation Failures when Checking Sidecar Connectivity
[ https://issues.apache.org/jira/browse/CASSANDRASC-86?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yifan Cai updated CASSANDRASC-86: - Reviewers: Yifan Cai Status: Review In Progress (was: Patch Available) > Startup Validation Failures when Checking Sidecar Connectivity > -- > > Key: CASSANDRASC-86 > URL: https://issues.apache.org/jira/browse/CASSANDRASC-86 > Project: Sidecar for Apache Cassandra > Issue Type: Improvement > Components: Configuration >Reporter: Yuriy Semchyshyn >Assignee: Yuriy Semchyshyn >Priority: Normal > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > We have experienced repeated startup validation failures caused by Sidecar > health checks for some jobs with a large number of Spark executors. > These failures are likely caused by the thundering herd problem, and have > been so far worked around by disabling startup validations altogether. > In order to prevent them going forward, a random delay needs to be added > between retries of health checks in Sidecar client. > It is also worth increasing the overall timeout for Sidecar health checks > from current 30 seconds to 60 seconds. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRASC-86) Startup Validation Failures when Checking Sidecar Connectivity
[ https://issues.apache.org/jira/browse/CASSANDRASC-86?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuriy Semchyshyn updated CASSANDRASC-86: Authors: Yuriy Semchyshyn Test and Documentation Plan: N/A Status: Patch Available (was: Open) > Startup Validation Failures when Checking Sidecar Connectivity > -- > > Key: CASSANDRASC-86 > URL: https://issues.apache.org/jira/browse/CASSANDRASC-86 > Project: Sidecar for Apache Cassandra > Issue Type: Improvement > Components: Configuration >Reporter: Yuriy Semchyshyn >Priority: Normal > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > We have experienced repeated startup validation failures caused by Sidecar > health checks for some jobs with a large number of Spark executors. > These failures are likely caused by the thundering herd problem, and have > been so far worked around by disabling startup validations altogether. > In order to prevent them going forward, a random delay needs to be added > between retries of health checks in Sidecar client. > It is also worth increasing the overall timeout for Sidecar health checks > from current 30 seconds to 60 seconds. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRASC-86) Startup Validation Failures when Checking Sidecar Connectivity
[ https://issues.apache.org/jira/browse/CASSANDRASC-86?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuriy Semchyshyn updated CASSANDRASC-86: Change Category: Semantic Complexity: Normal Component/s: Configuration Status: Open (was: Triage Needed) > Startup Validation Failures when Checking Sidecar Connectivity > -- > > Key: CASSANDRASC-86 > URL: https://issues.apache.org/jira/browse/CASSANDRASC-86 > Project: Sidecar for Apache Cassandra > Issue Type: Improvement > Components: Configuration >Reporter: Yuriy Semchyshyn >Priority: Normal > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > We have experienced repeated startup validation failures caused by Sidecar > health checks for some jobs with a large number of Spark executors. > These failures are likely caused by the thundering herd problem, and have > been so far worked around by disabling startup validations altogether. > In order to prevent them going forward, a random delay needs to be added > between retries of health checks in Sidecar client. > It is also worth increasing the overall timeout for Sidecar health checks > from current 30 seconds to 60 seconds. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRASC-86) Startup Validation Failures when Checking Sidecar Connectivity
[ https://issues.apache.org/jira/browse/CASSANDRASC-86?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated CASSANDRASC-86: -- Labels: pull-request-available (was: ) > Startup Validation Failures when Checking Sidecar Connectivity > -- > > Key: CASSANDRASC-86 > URL: https://issues.apache.org/jira/browse/CASSANDRASC-86 > Project: Sidecar for Apache Cassandra > Issue Type: Improvement >Reporter: Yuriy Semchyshyn >Priority: Normal > Labels: pull-request-available > > We have experienced repeated startup validation failures caused by Sidecar > health checks for some jobs with a large number of Spark executors. > These failures are likely caused by the thundering herd problem, and have > been so far worked around by disabling startup validations altogether. > In order to prevent them going forward, a random delay needs to be added > between retries of health checks in Sidecar client. > It is also worth increasing the overall timeout for Sidecar health checks > from current 30 seconds to 60 seconds. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
