[jira] [Commented] (CASSANDRA-17142) Limit the maximum hints size per host
[ https://issues.apache.org/jira/browse/CASSANDRA-17142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17498338#comment-17498338 ] Yifan Cai commented on CASSANDRA-17142: --- Made a [ninja commit|https://github.com/apache/cassandra/commit/b83da7d978f1bae5f289f1561dd080d48dc4505b] to update the size unit from megabytes to mebibytes. > Limit the maximum hints size per host > - > > Key: CASSANDRA-17142 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17142 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Hints >Reporter: Yifan Cai >Assignee: Yifan Cai >Priority: Normal > Fix For: 4.1 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > The hints system defines a time window, i.e. max_hint_window_in_ms, to store > the hints. > It defines no limit on how much data can be kept during the time window. The > hints can grow excessively and make the node running out of disk. In such > scenario, the operators have to truncate the hints manually. > I'd propose that in addition to the conventional hints window, operators > should be able to define the maximum hints size per host, i.e. > max_hints_size_per_host_in_mb, to provide an another layer of protection. A > node stops to store hints for the down node whenever it reaches to the time > cap or the size cap. In order to not surprise the users, the config should be > disabled by default. It should also be configurable via JMX. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17142) Limit the maximum hints size per host
[ https://issues.apache.org/jira/browse/CASSANDRA-17142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17490577#comment-17490577 ] Yifan Cai commented on CASSANDRA-17142: --- Starting commit Result: pending ||Branch ||Source||Circle CI|| |Trunk|[Code|https://github.com/yifan-c/cassandra/tree/CASSANDRA-17142/trunk]|[CI|https://app.circleci.com/pipelines/github/yifan-c/cassandra?branch=CASSANDRA-17142%2Ftrunk=all]| > Limit the maximum hints size per host > - > > Key: CASSANDRA-17142 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17142 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Hints >Reporter: Yifan Cai >Assignee: Yifan Cai >Priority: Normal > Time Spent: 2h 10m > Remaining Estimate: 0h > > The hints system defines a time window, i.e. max_hint_window_in_ms, to store > the hints. > It defines no limit on how much data can be kept during the time window. The > hints can grow excessively and make the node running out of disk. In such > scenario, the operators have to truncate the hints manually. > I'd propose that in addition to the conventional hints window, operators > should be able to define the maximum hints size per host, i.e. > max_hints_size_per_host_in_mb, to provide an another layer of protection. A > node stops to store hints for the down node whenever it reaches to the time > cap or the size cap. In order to not surprise the users, the config should be > disabled by default. It should also be configurable via JMX. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17142) Limit the maximum hints size per host
[ https://issues.apache.org/jira/browse/CASSANDRA-17142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17490572#comment-17490572 ] Dinesh Joshi commented on CASSANDRA-17142: -- +1 > Limit the maximum hints size per host > - > > Key: CASSANDRA-17142 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17142 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Hints >Reporter: Yifan Cai >Assignee: Yifan Cai >Priority: Normal > Time Spent: 2h 10m > Remaining Estimate: 0h > > The hints system defines a time window, i.e. max_hint_window_in_ms, to store > the hints. > It defines no limit on how much data can be kept during the time window. The > hints can grow excessively and make the node running out of disk. In such > scenario, the operators have to truncate the hints manually. > I'd propose that in addition to the conventional hints window, operators > should be able to define the maximum hints size per host, i.e. > max_hints_size_per_host_in_mb, to provide an another layer of protection. A > node stops to store hints for the down node whenever it reaches to the time > cap or the size cap. In order to not surprise the users, the config should be > disabled by default. It should also be configurable via JMX. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17142) Limit the maximum hints size per host
[ https://issues.apache.org/jira/browse/CASSANDRA-17142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461140#comment-17461140 ] Francisco Guerrero commented on CASSANDRA-17142: +1 added a couple minor comments > Limit the maximum hints size per host > - > > Key: CASSANDRA-17142 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17142 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Hints >Reporter: Yifan Cai >Assignee: Yifan Cai >Priority: Normal > Time Spent: 1.5h > Remaining Estimate: 0h > > The hints system defines a time window, i.e. max_hint_window_in_ms, to store > the hints. > It defines no limit on how much data can be kept during the time window. The > hints can grow excessively and make the node running out of disk. In such > scenario, the operators have to truncate the hints manually. > I'd propose that in addition to the conventional hints window, operators > should be able to define the maximum hints size per host, i.e. > max_hints_size_per_host_in_mb, to provide an another layer of protection. A > node stops to store hints for the down node whenever it reaches to the time > cap or the size cap. In order to not surprise the users, the config should be > disabled by default. It should also be configurable via JMX. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17142) Limit the maximum hints size per host
[ https://issues.apache.org/jira/browse/CASSANDRA-17142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17459517#comment-17459517 ] Yifan Cai commented on CASSANDRA-17142: --- [~frankgh], thanks for reviewing. I just pushed a commit to address. > Limit the maximum hints size per host > - > > Key: CASSANDRA-17142 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17142 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Hints >Reporter: Yifan Cai >Assignee: Yifan Cai >Priority: Normal > Time Spent: 1h 10m > Remaining Estimate: 0h > > The hints system defines a time window, i.e. max_hint_window_in_ms, to store > the hints. > It defines no limit on how much data can be kept during the time window. The > hints can grow excessively and make the node running out of disk. In such > scenario, the operators have to truncate the hints manually. > I'd propose that in addition to the conventional hints window, operators > should be able to define the maximum hints size per host, i.e. > max_hints_size_per_host_in_mb, to provide an another layer of protection. A > node stops to store hints for the down node whenever it reaches to the time > cap or the size cap. In order to not surprise the users, the config should be > disabled by default. It should also be configurable via JMX. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17142) Limit the maximum hints size per host
[
https://issues.apache.org/jira/browse/CASSANDRA-17142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456703#comment-17456703
]
Yifan Cai commented on CASSANDRA-17142:
---
PR: https://github.com/apache/cassandra/pull/1357
CI:
https://app.circleci.com/pipelines/github/yifan-c/cassandra/288/workflows/566bde90-9b69-4743-9e12-a97d4a4c137a
(mostly green)
As mentioned in the description, the PR add the configuration,
{{max_hints_size_per_host_in_mb}}, to limit the total size of the hints per
host. It is off by default.
Thanks [~Ge] for bringing up the Guardrails framework. Read though the CEP
and the first merged prototype. I think ultimately it is a good fit as you
mentioned. But as of now, the implementation is client facing since its
foundation classes depends on ClientState and ClientWarn. It does not fit very
well with those system-internal limits. I can see Guardrails is approaching it
iteratively. With respect to this ticket, we do not have to block it. We can
merge it first and address the required refactoring along with the other
system-internal limits when Guardrails evolves.
> Limit the maximum hints size per host
> -
>
> Key: CASSANDRA-17142
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17142
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Hints
>Reporter: Yifan Cai
>Assignee: Yifan Cai
>Priority: Normal
>
> The hints system defines a time window, i.e. max_hint_window_in_ms, to store
> the hints.
> It defines no limit on how much data can be kept during the time window. The
> hints can grow excessively and make the node running out of disk. In such
> scenario, the operators have to truncate the hints manually.
> I'd propose that in addition to the conventional hints window, operators
> should be able to define the maximum hints size per host, i.e.
> max_hints_size_per_host_in_mb, to provide an another layer of protection. A
> node stops to store hints for the down node whenever it reaches to the time
> cap or the size cap. In order to not surprise the users, the config should be
> disabled by default. It should also be configurable via JMX.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17142) Limit the maximum hints size per host
[ https://issues.apache.org/jira/browse/CASSANDRA-17142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17449856#comment-17449856 ] Aleksandr Sorokoumov commented on CASSANDRA-17142: -- This change can fit nicely in the Guardrails framework, similarly to https://issues.apache.org/jira/browse/CASSANDRA-17150. > Limit the maximum hints size per host > - > > Key: CASSANDRA-17142 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17142 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Hints >Reporter: Yifan Cai >Assignee: Yifan Cai >Priority: Normal > > The hints system defines a time window, i.e. max_hint_window_in_ms, to store > the hints. > It defines no limit on how much data can be kept during the time window. The > hints can grow excessively and make the node running out of disk. In such > scenario, the operators have to truncate the hints manually. > I'd propose that in addition to the conventional hints window, operators > should be able to define the maximum hints size per host, i.e. > max_hints_size_per_host_in_mb, to provide an another layer of protection. A > node stops to store hints for the down node whenever it reaches to the time > cap or the size cap. In order to not surprise the users, the config should be > disabled by default. It should also be configurable via JMX. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
