[jira] [Commented] (CASSANDRA-17808) Optionally avoid hint transfer during decommission
[
https://issues.apache.org/jira/browse/CASSANDRA-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833973#comment-17833973
]
Paul Chandler commented on CASSANDRA-17808:
---
https://issues.apache.org/jira/browse/CASSANDRA-19525 has been created
> Optionally avoid hint transfer during decommission
> --
>
> Key: CASSANDRA-17808
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17808
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Hints
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
> Fix For: 5.0-alpha1, 5.0
>
> Attachments: cassandra-17808_4.0.patch.txt,
> cassandra-17808_4.1.patch.txt
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Both because they aren’t strictly necessary to maintain consistency, and
> because throttling induced by their rate-limiter (see
> {{hinted_handoff_throttle}}) may stall progress, transferring hints during
> decommission (specifically unbootstrap) rather than just pausing, disabling,
> and truncating them probably doesn’t make sense. The only other concern would
> be the BatchLog, which nominally depends on hint delivery to maintain its
> "guarantees". However, during BatchLog replay on unbootstrap,
> {{ReplayingBatch}} ignores batches older the gcgs anyway.
> Here's a proposal from [~aleksey] that might strike a reasonable balance:
> 1.) We continue to transfer hints by default during decommission, but at a
> higher rate. We could, for instance, stop having {{DispatchHintsTask}} divide
> its effective rate by the number of nodes in the cluster.
> {noformat}
> int nodesCount = Math.max(1,
> StorageService.instance.getTokenMetadata().getAllEndpoints().size() - 1);
> double throttleInBytes = DatabaseDescriptor.getHintedHandoffThrottleInKiB() *
> 1024.0 / nodesCount;
> this.rateLimiter = RateLimiter.create(throttleInBytes == 0 ? Double.MAX_VALUE
> : throttleInBytes);
> {noformat}
> 2.) We provide an option to simply avoid transferring hints during
> unbootstrap. Even this would only take the BatchLog from "best effort" to
> "slightly less effort" ;)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17808) Optionally avoid hint transfer during decommission
[
https://issues.apache.org/jira/browse/CASSANDRA-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833964#comment-17833964
]
Brandon Williams commented on CASSANDRA-17808:
--
[~paulchandler] please do, thanks.
> Optionally avoid hint transfer during decommission
> --
>
> Key: CASSANDRA-17808
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17808
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Hints
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
> Fix For: 5.0-alpha1, 5.0
>
> Attachments: cassandra-17808_4.0.patch.txt,
> cassandra-17808_4.1.patch.txt
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Both because they aren’t strictly necessary to maintain consistency, and
> because throttling induced by their rate-limiter (see
> {{hinted_handoff_throttle}}) may stall progress, transferring hints during
> decommission (specifically unbootstrap) rather than just pausing, disabling,
> and truncating them probably doesn’t make sense. The only other concern would
> be the BatchLog, which nominally depends on hint delivery to maintain its
> "guarantees". However, during BatchLog replay on unbootstrap,
> {{ReplayingBatch}} ignores batches older the gcgs anyway.
> Here's a proposal from [~aleksey] that might strike a reasonable balance:
> 1.) We continue to transfer hints by default during decommission, but at a
> higher rate. We could, for instance, stop having {{DispatchHintsTask}} divide
> its effective rate by the number of nodes in the cluster.
> {noformat}
> int nodesCount = Math.max(1,
> StorageService.instance.getTokenMetadata().getAllEndpoints().size() - 1);
> double throttleInBytes = DatabaseDescriptor.getHintedHandoffThrottleInKiB() *
> 1024.0 / nodesCount;
> this.rateLimiter = RateLimiter.create(throttleInBytes == 0 ? Double.MAX_VALUE
> : throttleInBytes);
> {noformat}
> 2.) We provide an option to simply avoid transferring hints during
> unbootstrap. Even this would only take the BatchLog from "best effort" to
> "slightly less effort" ;)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17808) Optionally avoid hint transfer during decommission
[
https://issues.apache.org/jira/browse/CASSANDRA-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833962#comment-17833962
]
Paul Chandler commented on CASSANDRA-17808:
---
[~brandon.williams] shall I do that, and update CHANGES.txt for the new ticket
number ?
> Optionally avoid hint transfer during decommission
> --
>
> Key: CASSANDRA-17808
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17808
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Hints
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
> Fix For: 5.0-alpha1, 5.0
>
> Attachments: cassandra-17808_4.0.patch.txt,
> cassandra-17808_4.1.patch.txt
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Both because they aren’t strictly necessary to maintain consistency, and
> because throttling induced by their rate-limiter (see
> {{hinted_handoff_throttle}}) may stall progress, transferring hints during
> decommission (specifically unbootstrap) rather than just pausing, disabling,
> and truncating them probably doesn’t make sense. The only other concern would
> be the BatchLog, which nominally depends on hint delivery to maintain its
> "guarantees". However, during BatchLog replay on unbootstrap,
> {{ReplayingBatch}} ignores batches older the gcgs anyway.
> Here's a proposal from [~aleksey] that might strike a reasonable balance:
> 1.) We continue to transfer hints by default during decommission, but at a
> higher rate. We could, for instance, stop having {{DispatchHintsTask}} divide
> its effective rate by the number of nodes in the cluster.
> {noformat}
> int nodesCount = Math.max(1,
> StorageService.instance.getTokenMetadata().getAllEndpoints().size() - 1);
> double throttleInBytes = DatabaseDescriptor.getHintedHandoffThrottleInKiB() *
> 1024.0 / nodesCount;
> this.rateLimiter = RateLimiter.create(throttleInBytes == 0 ? Double.MAX_VALUE
> : throttleInBytes);
> {noformat}
> 2.) We provide an option to simply avoid transferring hints during
> unbootstrap. Even this would only take the BatchLog from "best effort" to
> "slightly less effort" ;)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17808) Optionally avoid hint transfer during decommission
[
https://issues.apache.org/jira/browse/CASSANDRA-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833958#comment-17833958
]
Brandon Williams commented on CASSANDRA-17808:
--
For clarity we should probably make a new ticket for the backports.
> Optionally avoid hint transfer during decommission
> --
>
> Key: CASSANDRA-17808
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17808
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Hints
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
> Fix For: 5.0-alpha1, 5.0
>
> Attachments: cassandra-17808_4.0.patch.txt,
> cassandra-17808_4.1.patch.txt
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Both because they aren’t strictly necessary to maintain consistency, and
> because throttling induced by their rate-limiter (see
> {{hinted_handoff_throttle}}) may stall progress, transferring hints during
> decommission (specifically unbootstrap) rather than just pausing, disabling,
> and truncating them probably doesn’t make sense. The only other concern would
> be the BatchLog, which nominally depends on hint delivery to maintain its
> "guarantees". However, during BatchLog replay on unbootstrap,
> {{ReplayingBatch}} ignores batches older the gcgs anyway.
> Here's a proposal from [~aleksey] that might strike a reasonable balance:
> 1.) We continue to transfer hints by default during decommission, but at a
> higher rate. We could, for instance, stop having {{DispatchHintsTask}} divide
> its effective rate by the number of nodes in the cluster.
> {noformat}
> int nodesCount = Math.max(1,
> StorageService.instance.getTokenMetadata().getAllEndpoints().size() - 1);
> double throttleInBytes = DatabaseDescriptor.getHintedHandoffThrottleInKiB() *
> 1024.0 / nodesCount;
> this.rateLimiter = RateLimiter.create(throttleInBytes == 0 ? Double.MAX_VALUE
> : throttleInBytes);
> {noformat}
> 2.) We provide an option to simply avoid transferring hints during
> unbootstrap. Even this would only take the BatchLog from "best effort" to
> "slightly less effort" ;)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17808) Optionally avoid hint transfer during decommission
[
https://issues.apache.org/jira/browse/CASSANDRA-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833957#comment-17833957
]
Stefan Miklosovic commented on CASSANDRA-17808:
---
[~paulchandler] I can backport this for next week.
> Optionally avoid hint transfer during decommission
> --
>
> Key: CASSANDRA-17808
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17808
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Hints
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
> Fix For: 5.0-alpha1, 5.0
>
> Attachments: cassandra-17808_4.0.patch.txt,
> cassandra-17808_4.1.patch.txt
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Both because they aren’t strictly necessary to maintain consistency, and
> because throttling induced by their rate-limiter (see
> {{hinted_handoff_throttle}}) may stall progress, transferring hints during
> decommission (specifically unbootstrap) rather than just pausing, disabling,
> and truncating them probably doesn’t make sense. The only other concern would
> be the BatchLog, which nominally depends on hint delivery to maintain its
> "guarantees". However, during BatchLog replay on unbootstrap,
> {{ReplayingBatch}} ignores batches older the gcgs anyway.
> Here's a proposal from [~aleksey] that might strike a reasonable balance:
> 1.) We continue to transfer hints by default during decommission, but at a
> higher rate. We could, for instance, stop having {{DispatchHintsTask}} divide
> its effective rate by the number of nodes in the cluster.
> {noformat}
> int nodesCount = Math.max(1,
> StorageService.instance.getTokenMetadata().getAllEndpoints().size() - 1);
> double throttleInBytes = DatabaseDescriptor.getHintedHandoffThrottleInKiB() *
> 1024.0 / nodesCount;
> this.rateLimiter = RateLimiter.create(throttleInBytes == 0 ? Double.MAX_VALUE
> : throttleInBytes);
> {noformat}
> 2.) We provide an option to simply avoid transferring hints during
> unbootstrap. Even this would only take the BatchLog from "best effort" to
> "slightly less effort" ;)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17808) Optionally avoid hint transfer during decommission
[
https://issues.apache.org/jira/browse/CASSANDRA-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833948#comment-17833948
]
Paul Chandler commented on CASSANDRA-17808:
---
Hi [~maedhroz] I have added patches for 4.0 and 4.1 to the ticket. I am not
sure what the next step would be.
> Optionally avoid hint transfer during decommission
> --
>
> Key: CASSANDRA-17808
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17808
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Hints
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
> Fix For: 5.0-alpha1, 5.0
>
> Attachments: cassandra-17808_4.0.patch.txt,
> cassandra-17808_4.1.patch.txt
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Both because they aren’t strictly necessary to maintain consistency, and
> because throttling induced by their rate-limiter (see
> {{hinted_handoff_throttle}}) may stall progress, transferring hints during
> decommission (specifically unbootstrap) rather than just pausing, disabling,
> and truncating them probably doesn’t make sense. The only other concern would
> be the BatchLog, which nominally depends on hint delivery to maintain its
> "guarantees". However, during BatchLog replay on unbootstrap,
> {{ReplayingBatch}} ignores batches older the gcgs anyway.
> Here's a proposal from [~aleksey] that might strike a reasonable balance:
> 1.) We continue to transfer hints by default during decommission, but at a
> higher rate. We could, for instance, stop having {{DispatchHintsTask}} divide
> its effective rate by the number of nodes in the cluster.
> {noformat}
> int nodesCount = Math.max(1,
> StorageService.instance.getTokenMetadata().getAllEndpoints().size() - 1);
> double throttleInBytes = DatabaseDescriptor.getHintedHandoffThrottleInKiB() *
> 1024.0 / nodesCount;
> this.rateLimiter = RateLimiter.create(throttleInBytes == 0 ? Double.MAX_VALUE
> : throttleInBytes);
> {noformat}
> 2.) We provide an option to simply avoid transferring hints during
> unbootstrap. Even this would only take the BatchLog from "best effort" to
> "slightly less effort" ;)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17808) Optionally avoid hint transfer during decommission
[
https://issues.apache.org/jira/browse/CASSANDRA-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17831511#comment-17831511
]
Caleb Rackliffe commented on CASSANDRA-17808:
-
[~paulchandler] That would be fantastic. Thanks!
> Optionally avoid hint transfer during decommission
> --
>
> Key: CASSANDRA-17808
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17808
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Hints
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
> Fix For: 5.0-alpha1, 5.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Both because they aren’t strictly necessary to maintain consistency, and
> because throttling induced by their rate-limiter (see
> {{hinted_handoff_throttle}}) may stall progress, transferring hints during
> decommission (specifically unbootstrap) rather than just pausing, disabling,
> and truncating them probably doesn’t make sense. The only other concern would
> be the BatchLog, which nominally depends on hint delivery to maintain its
> "guarantees". However, during BatchLog replay on unbootstrap,
> {{ReplayingBatch}} ignores batches older the gcgs anyway.
> Here's a proposal from [~aleksey] that might strike a reasonable balance:
> 1.) We continue to transfer hints by default during decommission, but at a
> higher rate. We could, for instance, stop having {{DispatchHintsTask}} divide
> its effective rate by the number of nodes in the cluster.
> {noformat}
> int nodesCount = Math.max(1,
> StorageService.instance.getTokenMetadata().getAllEndpoints().size() - 1);
> double throttleInBytes = DatabaseDescriptor.getHintedHandoffThrottleInKiB() *
> 1024.0 / nodesCount;
> this.rateLimiter = RateLimiter.create(throttleInBytes == 0 ? Double.MAX_VALUE
> : throttleInBytes);
> {noformat}
> 2.) We provide an option to simply avoid transferring hints during
> unbootstrap. Even this would only take the BatchLog from "best effort" to
> "slightly less effort" ;)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17808) Optionally avoid hint transfer during decommission
[
https://issues.apache.org/jira/browse/CASSANDRA-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17831472#comment-17831472
]
Paul Chandler commented on CASSANDRA-17808:
---
Hi [~maedhroz], if you have not started on this, I have some free time tomorrow
to have a go at making a patch for 4.0 and 4.1
> Optionally avoid hint transfer during decommission
> --
>
> Key: CASSANDRA-17808
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17808
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Hints
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
> Fix For: 5.0-alpha1, 5.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Both because they aren’t strictly necessary to maintain consistency, and
> because throttling induced by their rate-limiter (see
> {{hinted_handoff_throttle}}) may stall progress, transferring hints during
> decommission (specifically unbootstrap) rather than just pausing, disabling,
> and truncating them probably doesn’t make sense. The only other concern would
> be the BatchLog, which nominally depends on hint delivery to maintain its
> "guarantees". However, during BatchLog replay on unbootstrap,
> {{ReplayingBatch}} ignores batches older the gcgs anyway.
> Here's a proposal from [~aleksey] that might strike a reasonable balance:
> 1.) We continue to transfer hints by default during decommission, but at a
> higher rate. We could, for instance, stop having {{DispatchHintsTask}} divide
> its effective rate by the number of nodes in the cluster.
> {noformat}
> int nodesCount = Math.max(1,
> StorageService.instance.getTokenMetadata().getAllEndpoints().size() - 1);
> double throttleInBytes = DatabaseDescriptor.getHintedHandoffThrottleInKiB() *
> 1024.0 / nodesCount;
> this.rateLimiter = RateLimiter.create(throttleInBytes == 0 ? Double.MAX_VALUE
> : throttleInBytes);
> {noformat}
> 2.) We provide an option to simply avoid transferring hints during
> unbootstrap. Even this would only take the BatchLog from "best effort" to
> "slightly less effort" ;)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17808) Optionally avoid hint transfer during decommission
[
https://issues.apache.org/jira/browse/CASSANDRA-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829659#comment-17829659
]
Caleb Rackliffe commented on CASSANDRA-17808:
-
Let me see if I can find some time to back-port this to 4.0 and 4.1 this week
or next...
> Optionally avoid hint transfer during decommission
> --
>
> Key: CASSANDRA-17808
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17808
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Hints
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
> Fix For: 5.0-alpha1, 5.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Both because they aren’t strictly necessary to maintain consistency, and
> because throttling induced by their rate-limiter (see
> {{hinted_handoff_throttle}}) may stall progress, transferring hints during
> decommission (specifically unbootstrap) rather than just pausing, disabling,
> and truncating them probably doesn’t make sense. The only other concern would
> be the BatchLog, which nominally depends on hint delivery to maintain its
> "guarantees". However, during BatchLog replay on unbootstrap,
> {{ReplayingBatch}} ignores batches older the gcgs anyway.
> Here's a proposal from [~aleksey] that might strike a reasonable balance:
> 1.) We continue to transfer hints by default during decommission, but at a
> higher rate. We could, for instance, stop having {{DispatchHintsTask}} divide
> its effective rate by the number of nodes in the cluster.
> {noformat}
> int nodesCount = Math.max(1,
> StorageService.instance.getTokenMetadata().getAllEndpoints().size() - 1);
> double throttleInBytes = DatabaseDescriptor.getHintedHandoffThrottleInKiB() *
> 1024.0 / nodesCount;
> this.rateLimiter = RateLimiter.create(throttleInBytes == 0 ? Double.MAX_VALUE
> : throttleInBytes);
> {noformat}
> 2.) We provide an option to simply avoid transferring hints during
> unbootstrap. Even this would only take the BatchLog from "best effort" to
> "slightly less effort" ;)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17808) Optionally avoid hint transfer during decommission
[
https://issues.apache.org/jira/browse/CASSANDRA-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829621#comment-17829621
]
Paul Chandler commented on CASSANDRA-17808:
---
[~smiklosovic] and [~maedhroz] I can see that there is a PR for this change in
4.1, but it has not been committed. Are there any plans to do that.?
We would like to port it to 4.0 as we are having the same problem for a large
4.0 cluster, so we would be able to provide a patch for 4.0, but I wanted to
check there were no issues with the 4.1 version first that are stopping it
being committed?
> Optionally avoid hint transfer during decommission
> --
>
> Key: CASSANDRA-17808
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17808
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Hints
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
> Fix For: 5.0-alpha1, 5.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Both because they aren’t strictly necessary to maintain consistency, and
> because throttling induced by their rate-limiter (see
> {{hinted_handoff_throttle}}) may stall progress, transferring hints during
> decommission (specifically unbootstrap) rather than just pausing, disabling,
> and truncating them probably doesn’t make sense. The only other concern would
> be the BatchLog, which nominally depends on hint delivery to maintain its
> "guarantees". However, during BatchLog replay on unbootstrap,
> {{ReplayingBatch}} ignores batches older the gcgs anyway.
> Here's a proposal from [~aleksey] that might strike a reasonable balance:
> 1.) We continue to transfer hints by default during decommission, but at a
> higher rate. We could, for instance, stop having {{DispatchHintsTask}} divide
> its effective rate by the number of nodes in the cluster.
> {noformat}
> int nodesCount = Math.max(1,
> StorageService.instance.getTokenMetadata().getAllEndpoints().size() - 1);
> double throttleInBytes = DatabaseDescriptor.getHintedHandoffThrottleInKiB() *
> 1024.0 / nodesCount;
> this.rateLimiter = RateLimiter.create(throttleInBytes == 0 ? Double.MAX_VALUE
> : throttleInBytes);
> {noformat}
> 2.) We provide an option to simply avoid transferring hints during
> unbootstrap. Even this would only take the BatchLog from "best effort" to
> "slightly less effort" ;)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17808) Optionally avoid hint transfer during decommission
[
https://issues.apache.org/jira/browse/CASSANDRA-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17740311#comment-17740311
]
Stefan Miklosovic commented on CASSANDRA-17808:
---
[~maedhroz] are you ok with backporting to 4.1? How is this ad-hoc addition of
features done? We just agree on that?
> Optionally avoid hint transfer during decommission
> --
>
> Key: CASSANDRA-17808
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17808
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Hints
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
> Fix For: 5.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Both because they aren’t strictly necessary to maintain consistency, and
> because throttling induced by their rate-limiter (see
> {{hinted_handoff_throttle}}) may stall progress, transferring hints during
> decommission (specifically unbootstrap) rather than just pausing, disabling,
> and truncating them probably doesn’t make sense. The only other concern would
> be the BatchLog, which nominally depends on hint delivery to maintain its
> "guarantees". However, during BatchLog replay on unbootstrap,
> {{ReplayingBatch}} ignores batches older the gcgs anyway.
> Here's a proposal from [~aleksey] that might strike a reasonable balance:
> 1.) We continue to transfer hints by default during decommission, but at a
> higher rate. We could, for instance, stop having {{DispatchHintsTask}} divide
> its effective rate by the number of nodes in the cluster.
> {noformat}
> int nodesCount = Math.max(1,
> StorageService.instance.getTokenMetadata().getAllEndpoints().size() - 1);
> double throttleInBytes = DatabaseDescriptor.getHintedHandoffThrottleInKiB() *
> 1024.0 / nodesCount;
> this.rateLimiter = RateLimiter.create(throttleInBytes == 0 ? Double.MAX_VALUE
> : throttleInBytes);
> {noformat}
> 2.) We provide an option to simply avoid transferring hints during
> unbootstrap. Even this would only take the BatchLog from "best effort" to
> "slightly less effort" ;)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17808) Optionally avoid hint transfer during decommission
[
https://issues.apache.org/jira/browse/CASSANDRA-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738105#comment-17738105
]
Maxim Muzafarov commented on CASSANDRA-17808:
-
[~maedhroz], [~smiklosovic] Hello, can we also do the same exercise and
cherry-pick this issue as we did for the virtual tables running queries? I
think this is a helpful flag for users.
I have prepared changes here:
https://github.com/apache/cassandra/pull/2448
The test results on Jenkins:
[https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/2522/]
And again, I'm a bit limited with getting CircleCi results for this, so help is
needed :)
> Optionally avoid hint transfer during decommission
> --
>
> Key: CASSANDRA-17808
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17808
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Hints
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
> Fix For: 5.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Both because they aren’t strictly necessary to maintain consistency, and
> because throttling induced by their rate-limiter (see
> {{hinted_handoff_throttle}}) may stall progress, transferring hints during
> decommission (specifically unbootstrap) rather than just pausing, disabling,
> and truncating them probably doesn’t make sense. The only other concern would
> be the BatchLog, which nominally depends on hint delivery to maintain its
> "guarantees". However, during BatchLog replay on unbootstrap,
> {{ReplayingBatch}} ignores batches older the gcgs anyway.
> Here's a proposal from [~aleksey] that might strike a reasonable balance:
> 1.) We continue to transfer hints by default during decommission, but at a
> higher rate. We could, for instance, stop having {{DispatchHintsTask}} divide
> its effective rate by the number of nodes in the cluster.
> {noformat}
> int nodesCount = Math.max(1,
> StorageService.instance.getTokenMetadata().getAllEndpoints().size() - 1);
> double throttleInBytes = DatabaseDescriptor.getHintedHandoffThrottleInKiB() *
> 1024.0 / nodesCount;
> this.rateLimiter = RateLimiter.create(throttleInBytes == 0 ? Double.MAX_VALUE
> : throttleInBytes);
> {noformat}
> 2.) We provide an option to simply avoid transferring hints during
> unbootstrap. Even this would only take the BatchLog from "best effort" to
> "slightly less effort" ;)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17808) Optionally avoid hint transfer during decommission
[
https://issues.apache.org/jira/browse/CASSANDRA-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17601914#comment-17601914
]
Stefan Miklosovic commented on CASSANDRA-17808:
---
+1
> Optionally avoid hint transfer during decommission
> --
>
> Key: CASSANDRA-17808
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17808
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Hints
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
> Fix For: 4.x
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Both because they aren’t strictly necessary to maintain consistency, and
> because throttling induced by their rate-limiter (see
> {{hinted_handoff_throttle}}) may stall progress, transferring hints during
> decommission (specifically unbootstrap) rather than just pausing, disabling,
> and truncating them probably doesn’t make sense. The only other concern would
> be the BatchLog, which nominally depends on hint delivery to maintain its
> "guarantees". However, during BatchLog replay on unbootstrap,
> {{ReplayingBatch}} ignores batches older the gcgs anyway.
> Here's a proposal from [~aleksey] that might strike a reasonable balance:
> 1.) We continue to transfer hints by default during decommission, but at a
> higher rate. We could, for instance, stop having {{DispatchHintsTask}} divide
> its effective rate by the number of nodes in the cluster.
> {noformat}
> int nodesCount = Math.max(1,
> StorageService.instance.getTokenMetadata().getAllEndpoints().size() - 1);
> double throttleInBytes = DatabaseDescriptor.getHintedHandoffThrottleInKiB() *
> 1024.0 / nodesCount;
> this.rateLimiter = RateLimiter.create(throttleInBytes == 0 ? Double.MAX_VALUE
> : throttleInBytes);
> {noformat}
> 2.) We provide an option to simply avoid transferring hints during
> unbootstrap. Even this would only take the BatchLog from "best effort" to
> "slightly less effort" ;)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17808) Optionally avoid hint transfer during decommission
[
https://issues.apache.org/jira/browse/CASSANDRA-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17601906#comment-17601906
]
Jon Meredith commented on CASSANDRA-17808:
--
+1
> Optionally avoid hint transfer during decommission
> --
>
> Key: CASSANDRA-17808
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17808
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Hints
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
> Fix For: 4.x
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Both because they aren’t strictly necessary to maintain consistency, and
> because throttling induced by their rate-limiter (see
> {{hinted_handoff_throttle}}) may stall progress, transferring hints during
> decommission (specifically unbootstrap) rather than just pausing, disabling,
> and truncating them probably doesn’t make sense. The only other concern would
> be the BatchLog, which nominally depends on hint delivery to maintain its
> "guarantees". However, during BatchLog replay on unbootstrap,
> {{ReplayingBatch}} ignores batches older the gcgs anyway.
> Here's a proposal from [~aleksey] that might strike a reasonable balance:
> 1.) We continue to transfer hints by default during decommission, but at a
> higher rate. We could, for instance, stop having {{DispatchHintsTask}} divide
> its effective rate by the number of nodes in the cluster.
> {noformat}
> int nodesCount = Math.max(1,
> StorageService.instance.getTokenMetadata().getAllEndpoints().size() - 1);
> double throttleInBytes = DatabaseDescriptor.getHintedHandoffThrottleInKiB() *
> 1024.0 / nodesCount;
> this.rateLimiter = RateLimiter.create(throttleInBytes == 0 ? Double.MAX_VALUE
> : throttleInBytes);
> {noformat}
> 2.) We provide an option to simply avoid transferring hints during
> unbootstrap. Even this would only take the BatchLog from "best effort" to
> "slightly less effort" ;)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17808) Optionally avoid hint transfer during decommission
[
https://issues.apache.org/jira/browse/CASSANDRA-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17601505#comment-17601505
]
Caleb Rackliffe commented on CASSANDRA-17808:
-
Going to try to get this into review, now that CASSANDRA-16679 is committed...
> Optionally avoid hint transfer during decommission
> --
>
> Key: CASSANDRA-17808
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17808
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Hints
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
> Fix For: 4.x
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Both because they aren’t strictly necessary to maintain consistency, and
> because throttling induced by their rate-limiter (see
> {{hinted_handoff_throttle}}) may stall progress, transferring hints during
> decommission (specifically unbootstrap) rather than just pausing, disabling,
> and truncating them probably doesn’t make sense. The only other concern would
> be the BatchLog, which nominally depends on hint delivery to maintain its
> "guarantees". However, during BatchLog replay on unbootstrap,
> {{ReplayingBatch}} ignores batches older the gcgs anyway.
> Here's a proposal from [~aleksey] that might strike a reasonable balance:
> 1.) We continue to transfer hints by default during decommission, but at a
> higher rate. We could, for instance, stop having {{DispatchHintsTask}} divide
> its effective rate by the number of nodes in the cluster.
> {noformat}
> int nodesCount = Math.max(1,
> StorageService.instance.getTokenMetadata().getAllEndpoints().size() - 1);
> double throttleInBytes = DatabaseDescriptor.getHintedHandoffThrottleInKiB() *
> 1024.0 / nodesCount;
> this.rateLimiter = RateLimiter.create(throttleInBytes == 0 ? Double.MAX_VALUE
> : throttleInBytes);
> {noformat}
> 2.) We provide an option to simply avoid transferring hints during
> unbootstrap. Even this would only take the BatchLog from "best effort" to
> "slightly less effort" ;)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17808) Optionally avoid hint transfer during decommission
[
https://issues.apache.org/jira/browse/CASSANDRA-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17599255#comment-17599255
]
Caleb Rackliffe commented on CASSANDRA-17808:
-
Looks like I have, in fact, found the cause of the flakes in CASSANDRA-16679.
Moving over to fix that issue, then will rebase and finalize this.
> Optionally avoid hint transfer during decommission
> --
>
> Key: CASSANDRA-17808
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17808
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Hints
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
> Fix For: 4.x
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Both because they aren’t strictly necessary to maintain consistency, and
> because throttling induced by their rate-limiter (see
> {{hinted_handoff_throttle}}) may stall progress, transferring hints during
> decommission (specifically unbootstrap) rather than just pausing, disabling,
> and truncating them probably doesn’t make sense. The only other concern would
> be the BatchLog, which nominally depends on hint delivery to maintain its
> "guarantees". However, during BatchLog replay on unbootstrap,
> {{ReplayingBatch}} ignores batches older the gcgs anyway.
> Here's a proposal from [~aleksey] that might strike a reasonable balance:
> 1.) We continue to transfer hints by default during decommission, but at a
> higher rate. We could, for instance, stop having {{DispatchHintsTask}} divide
> its effective rate by the number of nodes in the cluster.
> {noformat}
> int nodesCount = Math.max(1,
> StorageService.instance.getTokenMetadata().getAllEndpoints().size() - 1);
> double throttleInBytes = DatabaseDescriptor.getHintedHandoffThrottleInKiB() *
> 1024.0 / nodesCount;
> this.rateLimiter = RateLimiter.create(throttleInBytes == 0 ? Double.MAX_VALUE
> : throttleInBytes);
> {noformat}
> 2.) We provide an option to simply avoid transferring hints during
> unbootstrap. Even this would only take the BatchLog from "best effort" to
> "slightly less effort" ;)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17808) Optionally avoid hint transfer during decommission
[
https://issues.apache.org/jira/browse/CASSANDRA-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17599163#comment-17599163
]
Ekaterina Dimitrova commented on CASSANDRA-17808:
-
Reading back in CASSANDRA-16679, unfortunately the CircleCI logs are gone but I
noticed we were hitting the following issue:
{code:java}
If I remove node 4 never to get back up, I still can see in the logs of node 2
that node 4 goes down and then a connection is established between node 2 and
node4 and node 4 is marked immediately up in the logs of node 2...{code}
Did this got resolved somewhere down the road? :D That would be awesome
> Optionally avoid hint transfer during decommission
> --
>
> Key: CASSANDRA-17808
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17808
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Hints
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
> Fix For: 4.x
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Both because they aren’t strictly necessary to maintain consistency, and
> because throttling induced by their rate-limiter (see
> {{hinted_handoff_throttle}}) may stall progress, transferring hints during
> decommission (specifically unbootstrap) rather than just pausing, disabling,
> and truncating them probably doesn’t make sense. The only other concern would
> be the BatchLog, which nominally depends on hint delivery to maintain its
> "guarantees". However, during BatchLog replay on unbootstrap,
> {{ReplayingBatch}} ignores batches older the gcgs anyway.
> Here's a proposal from [~aleksey] that might strike a reasonable balance:
> 1.) We continue to transfer hints by default during decommission, but at a
> higher rate. We could, for instance, stop having {{DispatchHintsTask}} divide
> its effective rate by the number of nodes in the cluster.
> {noformat}
> int nodesCount = Math.max(1,
> StorageService.instance.getTokenMetadata().getAllEndpoints().size() - 1);
> double throttleInBytes = DatabaseDescriptor.getHintedHandoffThrottleInKiB() *
> 1024.0 / nodesCount;
> this.rateLimiter = RateLimiter.create(throttleInBytes == 0 ? Double.MAX_VALUE
> : throttleInBytes);
> {noformat}
> 2.) We provide an option to simply avoid transferring hints during
> unbootstrap. Even this would only take the BatchLog from "best effort" to
> "slightly less effort" ;)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17808) Optionally avoid hint transfer during decommission
[
https://issues.apache.org/jira/browse/CASSANDRA-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17599157#comment-17599157
]
Caleb Rackliffe commented on CASSANDRA-17808:
-
Did a 300-iteration run of `HintedHandoffAddRemoveNodesTest` w/ some minor
fixes to make checking for hint totals more tolerant:
https://app.circleci.com/pipelines/github/maedhroz/cassandra?branch=CASSANDRA-17808
Things look clean, but I'll probably do another run after making the rest of
the changes I need to make for this issue...
> Optionally avoid hint transfer during decommission
> --
>
> Key: CASSANDRA-17808
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17808
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Hints
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
> Fix For: 4.x
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Both because they aren’t strictly necessary to maintain consistency, and
> because throttling induced by their rate-limiter (see
> {{hinted_handoff_throttle}}) may stall progress, transferring hints during
> decommission (specifically unbootstrap) rather than just pausing, disabling,
> and truncating them probably doesn’t make sense. The only other concern would
> be the BatchLog, which nominally depends on hint delivery to maintain its
> "guarantees". However, during BatchLog replay on unbootstrap,
> {{ReplayingBatch}} ignores batches older the gcgs anyway.
> Here's a proposal from [~aleksey] that might strike a reasonable balance:
> 1.) We continue to transfer hints by default during decommission, but at a
> higher rate. We could, for instance, stop having {{DispatchHintsTask}} divide
> its effective rate by the number of nodes in the cluster.
> {noformat}
> int nodesCount = Math.max(1,
> StorageService.instance.getTokenMetadata().getAllEndpoints().size() - 1);
> double throttleInBytes = DatabaseDescriptor.getHintedHandoffThrottleInKiB() *
> 1024.0 / nodesCount;
> this.rateLimiter = RateLimiter.create(throttleInBytes == 0 ? Double.MAX_VALUE
> : throttleInBytes);
> {noformat}
> 2.) We provide an option to simply avoid transferring hints during
> unbootstrap. Even this would only take the BatchLog from "best effort" to
> "slightly less effort" ;)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17808) Optionally avoid hint transfer during decommission
[
https://issues.apache.org/jira/browse/CASSANDRA-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17599071#comment-17599071
]
Caleb Rackliffe commented on CASSANDRA-17808:
-
I may try to address CASSANDRA-16679 while I'm in this corner of the codebase...
> Optionally avoid hint transfer during decommission
> --
>
> Key: CASSANDRA-17808
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17808
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Hints
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
> Fix For: 4.x
>
>
> Both because they aren’t strictly necessary to maintain consistency, and
> because throttling induced by their rate-limiter (see
> {{hinted_handoff_throttle}}) may stall progress, transferring hints during
> decommission (specifically unbootstrap) rather than just pausing, disabling,
> and truncating them probably doesn’t make sense. The only other concern would
> be the BatchLog, which nominally depends on hint delivery to maintain its
> "guarantees". However, during BatchLog replay on unbootstrap,
> {{ReplayingBatch}} ignores batches older the gcgs anyway.
> Here's a proposal from [~aleksey] that might strike a reasonable balance:
> 1.) We continue to transfer hints by default during decommission, but at a
> higher rate. We could, for instance, stop having {{DispatchHintsTask}} divide
> its effective rate by the number of nodes in the cluster.
> {noformat}
> int nodesCount = Math.max(1,
> StorageService.instance.getTokenMetadata().getAllEndpoints().size() - 1);
> double throttleInBytes = DatabaseDescriptor.getHintedHandoffThrottleInKiB() *
> 1024.0 / nodesCount;
> this.rateLimiter = RateLimiter.create(throttleInBytes == 0 ? Double.MAX_VALUE
> : throttleInBytes);
> {noformat}
> 2.) We provide an option to simply avoid transferring hints during
> unbootstrap. Even this would only take the BatchLog from "best effort" to
> "slightly less effort" ;)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
