[jira] [Commented] (CASSANDRA-16290) Consistency can be violated when bootstrap or decommission is resumed after node restart
[
https://issues.apache.org/jira/browse/CASSANDRA-16290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17756779#comment-17756779
]
Bartlomiej commented on CASSANDRA-16290:
I tested it "manually", I have two nodes locally (what took me most of the time
to setup this scenario), so yes, that change clears transferred ranges table
(both legacy and v2) after startup. I will write tests (unit or dtest) and will
be back :)
> Consistency can be violated when bootstrap or decommission is resumed after
> node restart
>
>
> Key: CASSANDRA-16290
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16290
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Bootstrap and Decommission
>Reporter: Paulo Motta
>Assignee: Bartlomiej
>Priority: Normal
> Labels: lhf
> Fix For: 3.0.x, 3.11.x, 4.0.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Since CASSANDRA-12008, successfully transferred ranges during decommission
> are saved on the {{system.transferred_ranges}} table. This allow skipping
> ranges already transferred when a failed decommission is retried with
> {{nodetool decommission}}.
> If instead of resuming the decommission, an operator restarts the node, waits
> N minutes and then performs a new decommission, the previously transferred
> ranges will be skipped during streaming, and any writes received by the
> decommissioned node during these N minutes will not be replicated to the new
> range owner, what violates consistency.
> This issue is analogous to the issue mentioned [on this
> comment|https://issues.apache.org/jira/browse/CASSANDRA-8838?focusedCommentId=16900234&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16900234]
> for resumable bootstrap (CASSANDRA-8838).
> In order to prevent consistency violations we should clear the
> {{system.transferred_ranges}} state during node restart, and maybe a system
> property to disable it. While we're at this, we should change the default of
> {{-Dcassandra.reset_bootstrap_progress}} to {{true}} to clear the
> {{system.available_ranges}} state by default when a bootstrapping node is
> restarted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16290) Consistency can be violated when bootstrap or decommission is resumed after node restart
[
https://issues.apache.org/jira/browse/CASSANDRA-16290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17756771#comment-17756771
]
Bartlomiej commented on CASSANDRA-16290:
sure [~smiklosovic] - I will try to test it as other properties are tested,
just wanted to ensure I go in the right direction :)
> Consistency can be violated when bootstrap or decommission is resumed after
> node restart
>
>
> Key: CASSANDRA-16290
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16290
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Bootstrap and Decommission
>Reporter: Paulo Motta
>Assignee: Bartlomiej
>Priority: Normal
> Labels: lhf
> Fix For: 3.0.x, 3.11.x, 4.0.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Since CASSANDRA-12008, successfully transferred ranges during decommission
> are saved on the {{system.transferred_ranges}} table. This allow skipping
> ranges already transferred when a failed decommission is retried with
> {{nodetool decommission}}.
> If instead of resuming the decommission, an operator restarts the node, waits
> N minutes and then performs a new decommission, the previously transferred
> ranges will be skipped during streaming, and any writes received by the
> decommissioned node during these N minutes will not be replicated to the new
> range owner, what violates consistency.
> This issue is analogous to the issue mentioned [on this
> comment|https://issues.apache.org/jira/browse/CASSANDRA-8838?focusedCommentId=16900234&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16900234]
> for resumable bootstrap (CASSANDRA-8838).
> In order to prevent consistency violations we should clear the
> {{system.transferred_ranges}} state during node restart, and maybe a system
> property to disable it. While we're at this, we should change the default of
> {{-Dcassandra.reset_bootstrap_progress}} to {{true}} to clear the
> {{system.available_ranges}} state by default when a bootstrapping node is
> restarted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16290) Consistency can be violated when bootstrap or decommission is resumed after node restart
[
https://issues.apache.org/jira/browse/CASSANDRA-16290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17756713#comment-17756713
]
Bartlomiej commented on CASSANDRA-16290:
Hi,
I played around this task and wanted to ask if the direction is correct ( pr
[https://github.com/apache/cassandra/pull/2614/files] ).
> Consistency can be violated when bootstrap or decommission is resumed after
> node restart
>
>
> Key: CASSANDRA-16290
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16290
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Bootstrap and Decommission
>Reporter: Paulo Motta
>Assignee: Bartlomiej
>Priority: Normal
> Labels: lhf
> Fix For: 3.0.x, 3.11.x, 4.0.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Since CASSANDRA-12008, successfully transferred ranges during decommission
> are saved on the {{system.transferred_ranges}} table. This allow skipping
> ranges already transferred when a failed decommission is retried with
> {{nodetool decommission}}.
> If instead of resuming the decommission, an operator restarts the node, waits
> N minutes and then performs a new decommission, the previously transferred
> ranges will be skipped during streaming, and any writes received by the
> decommissioned node during these N minutes will not be replicated to the new
> range owner, what violates consistency.
> This issue is analogous to the issue mentioned [on this
> comment|https://issues.apache.org/jira/browse/CASSANDRA-8838?focusedCommentId=16900234&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16900234]
> for resumable bootstrap (CASSANDRA-8838).
> In order to prevent consistency violations we should clear the
> {{system.transferred_ranges}} state during node restart, and maybe a system
> property to disable it. While we're at this, we should change the default of
> {{-Dcassandra.reset_bootstrap_progress}} to {{true}} to clear the
> {{system.available_ranges}} state by default when a bootstrapping node is
> restarted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16290) Consistency can be violated when bootstrap or decommission is resumed after node restart
[
https://issues.apache.org/jira/browse/CASSANDRA-16290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17748301#comment-17748301
]
Bartlomiej commented on CASSANDRA-16290:
Hi,
I would like to try implement this (hope it will not overwhelm me :D ). Can I
assign myself ?
> Consistency can be violated when bootstrap or decommission is resumed after
> node restart
>
>
> Key: CASSANDRA-16290
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16290
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Bootstrap and Decommission
>Reporter: Paulo Motta
>Priority: Normal
> Labels: lhf
> Fix For: 3.0.x, 3.11.x, 4.0.x
>
>
> Since CASSANDRA-12008, successfully transferred ranges during decommission
> are saved on the {{system.transferred_ranges}} table. This allow skipping
> ranges already transferred when a failed decommission is retried with
> {{nodetool decommission}}.
> If instead of resuming the decommission, an operator restarts the node, waits
> N minutes and then performs a new decommission, the previously transferred
> ranges will be skipped during streaming, and any writes received by the
> decommissioned node during these N minutes will not be replicated to the new
> range owner, what violates consistency.
> This issue is analogous to the issue mentioned [on this
> comment|https://issues.apache.org/jira/browse/CASSANDRA-8838?focusedCommentId=16900234&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16900234]
> for resumable bootstrap (CASSANDRA-8838).
> In order to prevent consistency violations we should clear the
> {{system.transferred_ranges}} state during node restart, and maybe a system
> property to disable it. While we're at this, we should change the default of
> {{-Dcassandra.reset_bootstrap_progress}} to {{true}} to clear the
> {{system.available_ranges}} state by default when a bootstrapping node is
> restarted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18298) Remove WaitingOnFreeMemtableSpace metric
[ https://issues.apache.org/jira/browse/CASSANDRA-18298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17745851#comment-17745851 ] Bartlomiej commented on CASSANDRA-18298: sorry for that :/ I didn't expect this logic can be spread so much :) right now all tests from stats package pass :) > Remove WaitingOnFreeMemtableSpace metric > > > Key: CASSANDRA-18298 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18298 > Project: Cassandra > Issue Type: Task > Components: Observability/Metrics >Reporter: Paulo Motta >Assignee: Bartlomiej >Priority: Normal > Labels: lhf > Fix For: 5.x > > Attachments: diff.txt > > Time Spent: 10m > Remaining Estimate: 0h > > It doesn't seem like this metric is [ever > updated|https://github.com/apache/cassandra/search?q=WaitingOnFreeMemtableSpace], > or ever was. > While we're at it we should remove any other unused metric. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18298) Remove WaitingOnFreeMemtableSpace metric
[
https://issues.apache.org/jira/browse/CASSANDRA-18298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17745639#comment-17745639
]
Bartlomiej commented on CASSANDRA-18298:
Ok, I think should be better now :) I have to say it was pretty tricky to
figure out what is going on because I was running this test from Intellij, and
output of this is not the same as
{code:java}
ant test -Dtest.name=TableStatsTest {code}
Can I ask you [~brandon.williams] for running CI once again ?
> Remove WaitingOnFreeMemtableSpace metric
>
>
> Key: CASSANDRA-18298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18298
> Project: Cassandra
> Issue Type: Task
> Components: Observability/Metrics
>Reporter: Paulo Motta
>Assignee: Bartlomiej
>Priority: Normal
> Labels: lhf
> Fix For: 5.x
>
> Attachments: diff.txt
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It doesn't seem like this metric is [ever
> updated|https://github.com/apache/cassandra/search?q=WaitingOnFreeMemtableSpace],
> or ever was.
> While we're at it we should remove any other unused metric.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18298) Remove WaitingOnFreeMemtableSpace metric
[ https://issues.apache.org/jira/browse/CASSANDRA-18298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17744916#comment-17744916 ] Bartlomiej commented on CASSANDRA-18298: yeah, there is still a property droppedMutations in TableStatsTest but honestly I didn't correlate it with metric DroppedMutations :) I will fix it, thanks for running tests. > Remove WaitingOnFreeMemtableSpace metric > > > Key: CASSANDRA-18298 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18298 > Project: Cassandra > Issue Type: Task > Components: Observability/Metrics >Reporter: Paulo Motta >Assignee: Bartlomiej >Priority: Normal > Labels: lhf > Fix For: 5.x > > Attachments: diff.txt > > Time Spent: 10m > Remaining Estimate: 0h > > It doesn't seem like this metric is [ever > updated|https://github.com/apache/cassandra/search?q=WaitingOnFreeMemtableSpace], > or ever was. > While we're at it we should remove any other unused metric. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18298) Remove WaitingOnFreeMemtableSpace metric
[ https://issues.apache.org/jira/browse/CASSANDRA-18298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17744816#comment-17744816 ] Bartlomiej commented on CASSANDRA-18298: Hi, I created initial PR [https://github.com/apache/cassandra/pull/2500] - can you please run tests in CI ? atm it takes ages on my computer to do that :D > Remove WaitingOnFreeMemtableSpace metric > > > Key: CASSANDRA-18298 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18298 > Project: Cassandra > Issue Type: Task > Components: Observability/Metrics >Reporter: Paulo Motta >Priority: Normal > Labels: lhf > Attachments: diff.txt > > Time Spent: 10m > Remaining Estimate: 0h > > It doesn't seem like this metric is [ever > updated|https://github.com/apache/cassandra/search?q=WaitingOnFreeMemtableSpace], > or ever was. > While we're at it we should remove any other unused metric. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18298) Remove WaitingOnFreeMemtableSpace metric
[ https://issues.apache.org/jira/browse/CASSANDRA-18298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17743992#comment-17743992 ] Bartlomiej commented on CASSANDRA-18298: thanks [~brandon.williams] - I will do that. > Remove WaitingOnFreeMemtableSpace metric > > > Key: CASSANDRA-18298 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18298 > Project: Cassandra > Issue Type: Task > Components: Observability/Metrics >Reporter: Paulo Motta >Priority: Normal > Labels: lhf > Attachments: diff.txt > > > It doesn't seem like this metric is [ever > updated|https://github.com/apache/cassandra/search?q=WaitingOnFreeMemtableSpace], > or ever was. > While we're at it we should remove any other unused metric. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18298) Remove WaitingOnFreeMemtableSpace metric
[ https://issues.apache.org/jira/browse/CASSANDRA-18298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bartlomiej updated CASSANDRA-18298: --- Attachment: diff.txt > Remove WaitingOnFreeMemtableSpace metric > > > Key: CASSANDRA-18298 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18298 > Project: Cassandra > Issue Type: Task > Components: Observability/Metrics >Reporter: Paulo Motta >Priority: Normal > Labels: lhf > Attachments: diff.txt > > > It doesn't seem like this metric is [ever > updated|https://github.com/apache/cassandra/search?q=WaitingOnFreeMemtableSpace], > or ever was. > While we're at it we should remove any other unused metric. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18298) Remove WaitingOnFreeMemtableSpace metric
[ https://issues.apache.org/jira/browse/CASSANDRA-18298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17743988#comment-17743988 ] Bartlomiej commented on CASSANDRA-18298: Hi, I would like to work on this, just a few questions: I removed waitingOnFreeMemtableSpace metric and now there is a need for change in JMXCompatabilityTest, my question is if the regression should be handled as in diff i provided (so, I don't want to regenerate entire cassandra-X.X-jmx.yaml, just exclude removed, breaking change metric and annotate with relevant Jira ticket). [^diff.txt] > Remove WaitingOnFreeMemtableSpace metric > > > Key: CASSANDRA-18298 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18298 > Project: Cassandra > Issue Type: Task > Components: Observability/Metrics >Reporter: Paulo Motta >Priority: Normal > Labels: lhf > Attachments: diff.txt > > > It doesn't seem like this metric is [ever > updated|https://github.com/apache/cassandra/search?q=WaitingOnFreeMemtableSpace], > or ever was. > While we're at it we should remove any other unused metric. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18298) Remove WaitingOnFreeMemtableSpace metric
[ https://issues.apache.org/jira/browse/CASSANDRA-18298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bartlomiej updated CASSANDRA-18298: --- Attachment: diff.txt > Remove WaitingOnFreeMemtableSpace metric > > > Key: CASSANDRA-18298 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18298 > Project: Cassandra > Issue Type: Task > Components: Observability/Metrics >Reporter: Paulo Motta >Priority: Normal > Labels: lhf > > It doesn't seem like this metric is [ever > updated|https://github.com/apache/cassandra/search?q=WaitingOnFreeMemtableSpace], > or ever was. > While we're at it we should remove any other unused metric. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18298) Remove WaitingOnFreeMemtableSpace metric
[ https://issues.apache.org/jira/browse/CASSANDRA-18298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bartlomiej updated CASSANDRA-18298: --- Attachment: (was: diff.txt) > Remove WaitingOnFreeMemtableSpace metric > > > Key: CASSANDRA-18298 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18298 > Project: Cassandra > Issue Type: Task > Components: Observability/Metrics >Reporter: Paulo Motta >Priority: Normal > Labels: lhf > > It doesn't seem like this metric is [ever > updated|https://github.com/apache/cassandra/search?q=WaitingOnFreeMemtableSpace], > or ever was. > While we're at it we should remove any other unused metric. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-17189) Guardrail for page size
[
https://issues.apache.org/jira/browse/CASSANDRA-17189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459501#comment-17459501
]
Bartlomiej edited comment on CASSANDRA-17189 at 12/14/21, 9:32 PM:
---
Thanks a lot [~adelapena] for help with tests (especially with those
executeWithPaging method - it would take me ages to make it working like this).
I also applied your changes (sorry for not applying code-formatting before - I
just realized that Intellij config has it out-of-the-box). Here is the PR
[https://github.com/apache/cassandra/pull/1361] (so it will be easier to
comment :) )
once again thanks for all your help ;)
was (Author: bkowalczyyk):
Thanks a lot [~adelapena] for help with tests (especially with those
executeWithPaging method - it would take me ages to make it working like this).
I also applied your changes (sorry for not applying code-formatting - I just
realized that Intellij config has it out-of-the-box). Here is the PR
[https://github.com/apache/cassandra/pull/1361] (so it will be easier to
comment :) )
once again thanks for all your help ;)
> Guardrail for page size
> ---
>
> Key: CASSANDRA-17189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17189
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Guardrails
>Reporter: Andres de la Peña
>Assignee: Bartlomiej
>Priority: Normal
> Labels: AdventCalendar2021, lhf
> Fix For: 4.1
>
> Attachments: CASSANDRA-17189-trunk.diff
>
>
> Add guardrail limiting the query page size, for example:
> {code}
> # Guardrail to warn about or reject page sizes greater than threshold.
> # The two thresholds default to -1 to disable.
> page_size:
> warn_threshold: -1
> abort_threshold: -1
> {code}
> Initially this can be based on the specified number of rows used as page
> size, although it would be ideal to also limit the actual size in bytes of
> the returned pages.
> +Additional information for newcomers:+
> # Add the configuration for the new guardrail on page size in the guardrails
> section of cassandra.yaml.
> # Add a getPageSize method in GuardrailsConfig returning a Threshold.Config
> object
> # Implement that method in GuardrailsOptions, which is the default yaml-based
> implementation of GuardrailsConfig
> # Add a Threshold guardrail named pageSize in Guardrails, using the
> previously created config
> # Define JMX-friendly getters and setters for the previously created config
> in GuardrailsMBean
> # Implement the JMX-friendly getters and setters in Guardrails
> # Now that we have the guardrail ready, it’s time to use it. We should search
> for a place to invoke the Guardrails.pageSize#guard method with the page size
> that each query is going to use. The DataLimits#forPaging methods look like
> good candidates for this.
> # Finally, add some tests for the new guardrail. Given that the new guardrail
> is a Threshold, our new test should probably extend ThresholdTester.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-17189) Guardrail for page size
[
https://issues.apache.org/jira/browse/CASSANDRA-17189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459501#comment-17459501
]
Bartlomiej edited comment on CASSANDRA-17189 at 12/14/21, 9:32 PM:
---
Thanks a lot [~adelapena] for help with tests (especially with those
executeWithPaging method - it would take me ages to make it working like this).
I also applied your changes (sorry for not applying code-formatting - I just
realized that Intellij config has it out-of-the-box). Here is the PR
[https://github.com/apache/cassandra/pull/1361] (so it will be easier to
comment :) )
once again thanks for all your help ;)
was (Author: bkowalczyyk):
Thanks a lot [~adelapena] for help with tests (especially with those
executeWithPaging method - it would take me ages to make it working like this).
I also applied your changes (sorry for not applying code-formatting - I just
realized that Intellij config has it out-of-the-box). Here is the PR
[https://github.com/apache/cassandra/pull/1361] (so it will be easier to
comment :) )
once again thanks for all your help ;)
> Guardrail for page size
> ---
>
> Key: CASSANDRA-17189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17189
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Guardrails
>Reporter: Andres de la Peña
>Assignee: Bartlomiej
>Priority: Normal
> Labels: AdventCalendar2021, lhf
> Fix For: 4.1
>
> Attachments: CASSANDRA-17189-trunk.diff
>
>
> Add guardrail limiting the query page size, for example:
> {code}
> # Guardrail to warn about or reject page sizes greater than threshold.
> # The two thresholds default to -1 to disable.
> page_size:
> warn_threshold: -1
> abort_threshold: -1
> {code}
> Initially this can be based on the specified number of rows used as page
> size, although it would be ideal to also limit the actual size in bytes of
> the returned pages.
> +Additional information for newcomers:+
> # Add the configuration for the new guardrail on page size in the guardrails
> section of cassandra.yaml.
> # Add a getPageSize method in GuardrailsConfig returning a Threshold.Config
> object
> # Implement that method in GuardrailsOptions, which is the default yaml-based
> implementation of GuardrailsConfig
> # Add a Threshold guardrail named pageSize in Guardrails, using the
> previously created config
> # Define JMX-friendly getters and setters for the previously created config
> in GuardrailsMBean
> # Implement the JMX-friendly getters and setters in Guardrails
> # Now that we have the guardrail ready, it’s time to use it. We should search
> for a place to invoke the Guardrails.pageSize#guard method with the page size
> that each query is going to use. The DataLimits#forPaging methods look like
> good candidates for this.
> # Finally, add some tests for the new guardrail. Given that the new guardrail
> is a Threshold, our new test should probably extend ThresholdTester.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17189) Guardrail for page size
[
https://issues.apache.org/jira/browse/CASSANDRA-17189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459501#comment-17459501
]
Bartlomiej commented on CASSANDRA-17189:
Thanks a lot [~adelapena] for help with tests (especially with those
executeWithPaging method - it would take me ages to make it working like this).
I also applied your changes (sorry for not applying code-formatting - I just
realized that Intellij config has it out-of-the-box). Here is the PR
[https://github.com/apache/cassandra/pull/1361] (so it will be easier to
comment :) )
once again thanks for all your help ;)
> Guardrail for page size
> ---
>
> Key: CASSANDRA-17189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17189
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Guardrails
>Reporter: Andres de la Peña
>Assignee: Bartlomiej
>Priority: Normal
> Labels: AdventCalendar2021, lhf
> Fix For: 4.1
>
> Attachments: CASSANDRA-17189-trunk.diff
>
>
> Add guardrail limiting the query page size, for example:
> {code}
> # Guardrail to warn about or reject page sizes greater than threshold.
> # The two thresholds default to -1 to disable.
> page_size:
> warn_threshold: -1
> abort_threshold: -1
> {code}
> Initially this can be based on the specified number of rows used as page
> size, although it would be ideal to also limit the actual size in bytes of
> the returned pages.
> +Additional information for newcomers:+
> # Add the configuration for the new guardrail on page size in the guardrails
> section of cassandra.yaml.
> # Add a getPageSize method in GuardrailsConfig returning a Threshold.Config
> object
> # Implement that method in GuardrailsOptions, which is the default yaml-based
> implementation of GuardrailsConfig
> # Add a Threshold guardrail named pageSize in Guardrails, using the
> previously created config
> # Define JMX-friendly getters and setters for the previously created config
> in GuardrailsMBean
> # Implement the JMX-friendly getters and setters in Guardrails
> # Now that we have the guardrail ready, it’s time to use it. We should search
> for a place to invoke the Guardrails.pageSize#guard method with the page size
> that each query is going to use. The DataLimits#forPaging methods look like
> good candidates for this.
> # Finally, add some tests for the new guardrail. Given that the new guardrail
> is a Threshold, our new test should probably extend ThresholdTester.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-17181) SchemaCQLHelperTest methods can be simplified
[
https://issues.apache.org/jira/browse/CASSANDRA-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bartlomiej updated CASSANDRA-17181:
---
Attachment: CASSANDRA-17181-4.0.patch
> SchemaCQLHelperTest methods can be simplified
> -
>
> Key: CASSANDRA-17181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17181
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Snapshots
>Reporter: Benjamin Lerer
>Assignee: Bartlomiej
>Priority: Normal
> Labels: AdventCalendar2021, lhf
> Fix For: 4.0.x, 4.x
>
> Attachments: CASSANDRA-17181-4.0.patch
>
>
> {{SchemaCQLHelperTest}} is used during a snapshot to generate the
> {{schema.cql}} file. The methods accept the following paramaters:
> {{includeDroppedColumns}}, {{internals}} and {{ifNotExists}}.
> Those parameters are in practice always set to true by the calling code and
> therefore can be removed.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17181) SchemaCQLHelperTest methods can be simplified
[
https://issues.apache.org/jira/browse/CASSANDRA-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17458378#comment-17458378
]
Bartlomiej commented on CASSANDRA-17181:
Hi :)
I added comments you suggested (honestly I am not sure my comments make sense
:D )
{code:java}
@VisibleForTesting
public static String getTableMetadataAsCQL(TableMetadata metadata)
{
if (metadata.isView())
{
KeyspaceMetadata keyspaceMetadata =
Schema.instance.getKeyspaceMetadata(metadata.keyspace);
ViewMetadata viewMetadata =
keyspaceMetadata.views.get(metadata.name).orElse(null);
assert viewMetadata != null;
/*
* first argument(withInternals) indicates to include table metadata id
and clustering columns order,
* second argument(ifNotExists) instructs to include IF NOT EXISTS
statement withing creation statements.
*/
return viewMetadata.toCqlString(true, true);
}
/*
* With addition to withInternals and ifNotExists argruments,
includeDroppedColumns will include dropped
* columns as ALTER TABLE statements appended into the snapshot.
*/
return metadata.toCqlString(true, true, true);
} {code}
If you have any hints here, pls say.
{quote}
if a branch or a PR in your fork is more convenient
{quote}
sure, next time I will make a PR.
> SchemaCQLHelperTest methods can be simplified
> -
>
> Key: CASSANDRA-17181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17181
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Snapshots
>Reporter: Benjamin Lerer
>Assignee: Bartlomiej
>Priority: Normal
> Labels: AdventCalendar2021, lhf
> Fix For: 4.0.x, 4.x
>
>
> {{SchemaCQLHelperTest}} is used during a snapshot to generate the
> {{schema.cql}} file. The methods accept the following paramaters:
> {{includeDroppedColumns}}, {{internals}} and {{ifNotExists}}.
> Those parameters are in practice always set to true by the calling code and
> therefore can be removed.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-17181) SchemaCQLHelperTest methods can be simplified
[
https://issues.apache.org/jira/browse/CASSANDRA-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bartlomiej updated CASSANDRA-17181:
---
Attachment: (was: CASSANDRA-17181-4.0.patch)
> SchemaCQLHelperTest methods can be simplified
> -
>
> Key: CASSANDRA-17181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17181
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Snapshots
>Reporter: Benjamin Lerer
>Assignee: Bartlomiej
>Priority: Normal
> Labels: AdventCalendar2021, lhf
> Fix For: 4.0.x, 4.x
>
>
> {{SchemaCQLHelperTest}} is used during a snapshot to generate the
> {{schema.cql}} file. The methods accept the following paramaters:
> {{includeDroppedColumns}}, {{internals}} and {{ifNotExists}}.
> Those parameters are in practice always set to true by the calling code and
> therefore can be removed.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-17189) Guardrail for page size
[
https://issues.apache.org/jira/browse/CASSANDRA-17189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17457359#comment-17457359
]
Bartlomiej edited comment on CASSANDRA-17189 at 12/10/21, 11:27 PM:
{code:java}
Why would it crash?{code}
that was because, in forPaging method, state was null
{code:java}
public DataLimits forPaging(int pageSize)
{
Guardrails.pageSize.guard(pageSize, "?", null);
return new CQLLimits(pageSize, perPartitionLimit, isDistinct);
}{code}
so guard was also applied for super user
{code:java}
(state == null || state.isOrdinaryUser()); {code}
Now it is not the case anymore because in execute I am able to get state and
super users are not aborded :)
{code:java}
By the way, that SelectStatement#columnFamily method uses the old terminology
for tables, which were originally called column families. Maybe we can rename
the method to table(). {code}
as you suggested I changed two methods to table() :)
I also had to move guard call from *public execute()* to *private execute()*
because otherwise *executeInternal()* would stay without guard check or this
guard call would have be duplicated. Now, after reading the code, I understand
that all Select statements will be handled by that guard. Please say what you
think about my patch, if it is "lets say ok" I will write tests :)
Thanks !
was (Author: bkowalczyyk):
{code:java}
Why would it crash?{code}
that was because, in forPaging method, state was null
{code:java}
public DataLimits forPaging(int pageSize)
{
Guardrails.pageSize.guard(pageSize, "?", null);
return new CQLLimits(pageSize, perPartitionLimit, isDistinct);
}{code}
so guard was also applied for super user
{code:java}
(state == null || state.isOrdinaryUser()); {code}
Now it is not the case anymore because in execute I am able to get state and
super users are not aborded :)
{code:java}
By the way, that SelectStatement#columnFamily method uses the old terminology
for tables, which were originally called column families. Maybe we can rename
the method to table(). {code}
as you suggested I changed two methods to table() :)
I also had to move guard call from
[https://github.com/apache/cassandra/blob/e99a8da161ed599c1a22a853c9c7f9caf6c1eb79/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java#L236|public
execute()] to
[https://github.com/apache/cassandra/blob/e99a8da161ed599c1a22a853c9c7f9caf6c1eb79/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java#L296|private
execute()] because otherwise
[https://github.com/apache/cassandra/blob/e99a8da161ed599c1a22a853c9c7f9caf6c1eb79/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java#L443|executeInternal()]
would stay without guard check or this guard call would have be duplicated.
Now, after reading the code, I understand that all Select statements will be
handled by that guard. Please say what you think about my patch, if it is "lets
say ok" I will write tests :)
Thanks !
> Guardrail for page size
> ---
>
> Key: CASSANDRA-17189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17189
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Guardrails
>Reporter: Andres de la Peña
>Assignee: Bartlomiej
>Priority: Normal
> Labels: AdventCalendar2021, lhf
> Fix For: 4.1
>
> Attachments: CASSANDRA-17189-trunk.diff
>
>
> Add guardrail limiting the query page size, for example:
> {code}
> # Guardrail to warn about or reject page sizes greater than threshold.
> # The two thresholds default to -1 to disable.
> page_size:
> warn_threshold: -1
> abort_threshold: -1
> {code}
> Initially this can be based on the specified number of rows used as page
> size, although it would be ideal to also limit the actual size in bytes of
> the returned pages.
> +Additional information for newcomers:+
> # Add the configuration for the new guardrail on page size in the guardrails
> section of cassandra.yaml.
> # Add a getPageSize method in GuardrailsConfig returning a Threshold.Config
> object
> # Implement that method in GuardrailsOptions, which is the default yaml-based
> implementation of GuardrailsConfig
> # Add a Threshold guardrail named pageSize in Guardrails, using the
> previously created config
> # Define JMX-friendly getters and setters for the previously created config
> in GuardrailsMBean
> # Implement the JMX-friendly getters and setters in Guardrails
> # Now that we have the guardrail ready, it’s time to use it. We should search
> for a place to invoke the Guardrails.pageSize#guard method with the page size
> that each query is going to use. The DataLimits#forPaging methods look like
> good candidates for this.
> # Finally, add some tests for the new guardrail. Given that the new gu
[jira] [Comment Edited] (CASSANDRA-17189) Guardrail for page size
[
https://issues.apache.org/jira/browse/CASSANDRA-17189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17457359#comment-17457359
]
Bartlomiej edited comment on CASSANDRA-17189 at 12/10/21, 11:22 PM:
{code:java}
Why would it crash?{code}
that was because, in forPaging method, state was null
{code:java}
public DataLimits forPaging(int pageSize)
{
Guardrails.pageSize.guard(pageSize, "?", null);
return new CQLLimits(pageSize, perPartitionLimit, isDistinct);
}{code}
so guard was also applied for super user
{code:java}
(state == null || state.isOrdinaryUser()); {code}
Now it is not the case anymore because in execute I am able to get state and
super users are not aborded :)
{code:java}
By the way, that SelectStatement#columnFamily method uses the old terminology
for tables, which were originally called column families. Maybe we can rename
the method to table(). {code}
as you suggested I changed two methods to table() :)
I also had to move guard call from
[https://github.com/apache/cassandra/blob/e99a8da161ed599c1a22a853c9c7f9caf6c1eb79/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java#L236|public
execute()] to
[https://github.com/apache/cassandra/blob/e99a8da161ed599c1a22a853c9c7f9caf6c1eb79/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java#L296|private
execute()] because otherwise
[https://github.com/apache/cassandra/blob/e99a8da161ed599c1a22a853c9c7f9caf6c1eb79/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java#L443|executeInternal()]
would stay without guard check or this guard call would have be duplicated.
Now, after reading the code, I understand that all Select statements will be
handled by that guard. Please say what you think about my patch, if it is "lets
say ok" I will write tests :)
Thanks !
was (Author: bkowalczyyk):
{code:java}
Why would it crash?{code}
that was because, in forPaging method, state was null
{code:java}
public DataLimits forPaging(int pageSize)
{
Guardrails.pageSize.guard(pageSize, "?", null);
return new CQLLimits(pageSize, perPartitionLimit, isDistinct);
}{code}
so guard was also applied for super user
{code:java}
(state == null || state.isOrdinaryUser()); {code}
Now it is not the case anymore because in execute I am able to get state and
super users are not aborded :)
{code:java}
By the way, that SelectStatement#columnFamily method uses the old terminology
for tables, which were originally called column families. Maybe we can rename
the method to table(). {code}
as you suggested I changed two methods to table() :)
I also had to move guard call from
{code:java}
public ResultMessage.Rows execute() {code}
to
{code:java}
private ResultMessage.Rows execute(){code}
because otherwise
{code:java}
public ResultMessage.Rows executeInternal() {code}
would stay without guard check or this guard call would have be duplicated.
Now, after reading the code, I understand that all Select statements will be
handled by that guard. Please say what you think about my patch, if it is "lets
say ok" I will write tests :)
Thanks !
> Guardrail for page size
> ---
>
> Key: CASSANDRA-17189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17189
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Guardrails
>Reporter: Andres de la Peña
>Assignee: Bartlomiej
>Priority: Normal
> Labels: AdventCalendar2021, lhf
> Fix For: 4.1
>
> Attachments: CASSANDRA-17189-trunk.diff
>
>
> Add guardrail limiting the query page size, for example:
> {code}
> # Guardrail to warn about or reject page sizes greater than threshold.
> # The two thresholds default to -1 to disable.
> page_size:
> warn_threshold: -1
> abort_threshold: -1
> {code}
> Initially this can be based on the specified number of rows used as page
> size, although it would be ideal to also limit the actual size in bytes of
> the returned pages.
> +Additional information for newcomers:+
> # Add the configuration for the new guardrail on page size in the guardrails
> section of cassandra.yaml.
> # Add a getPageSize method in GuardrailsConfig returning a Threshold.Config
> object
> # Implement that method in GuardrailsOptions, which is the default yaml-based
> implementation of GuardrailsConfig
> # Add a Threshold guardrail named pageSize in Guardrails, using the
> previously created config
> # Define JMX-friendly getters and setters for the previously created config
> in GuardrailsMBean
> # Implement the JMX-friendly getters and setters in Guardrails
> # Now that we have the guardrail ready, it’s time to use it. We should search
> for a place to invoke the Guardrails.pageSize#guard method with the page size
> that each query is going to use. The DataLimits#forPaging method
[jira] [Comment Edited] (CASSANDRA-17189) Guardrail for page size
[
https://issues.apache.org/jira/browse/CASSANDRA-17189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17457359#comment-17457359
]
Bartlomiej edited comment on CASSANDRA-17189 at 12/10/21, 9:07 PM:
---
{code:java}
Why would it crash?{code}
that was because, in forPaging method, state was null
{code:java}
public DataLimits forPaging(int pageSize)
{
Guardrails.pageSize.guard(pageSize, "?", null);
return new CQLLimits(pageSize, perPartitionLimit, isDistinct);
}{code}
so guard was also applied for super user
{code:java}
(state == null || state.isOrdinaryUser()); {code}
Now it is not the case anymore because in execute I am able to get state and
super users are not aborded :)
{code:java}
By the way, that SelectStatement#columnFamily method uses the old terminology
for tables, which were originally called column families. Maybe we can rename
the method to table(). {code}
as you suggested I changed two methods to table() :)
I also had to move guard call from
{code:java}
public ResultMessage.Rows execute() {code}
to
{code:java}
private ResultMessage.Rows execute(){code}
because otherwise
{code:java}
public ResultMessage.Rows executeInternal() {code}
would stay without guard check or this guard call would have be duplicated.
Now, after reading the code, I understand that all Select statements will be
handled by that guard. Please say what you think about my patch, if it is "lets
say ok" I will write tests :)
Thanks !
was (Author: bkowalczyyk):
{code:java}
Why would it crash?{code}
that was because, in forPaging method, state was null
{code:java}
public DataLimits forPaging(int pageSize)
{
Guardrails.pageSize.guard(pageSize, "?", null);
return new CQLLimits(pageSize, perPartitionLimit, isDistinct);
}{code}
so guard was also applied for super user
{code:java}
(state == null || state.isOrdinaryUser()); {code}
Now it is not the case anymore because in execute I am able to get state and
super users are not aborded :)
{code:java}
By the way, that SelectStatement#columnFamily method uses the old terminology
for tables, which were originally called column families. Maybe we can rename
the method to table(). {code}
as you suggested I changed two methods to table() :)
I also had to move guard call from
{code:java}
public ResultMessage.Rows execute() {code}
to
{code:java}
private ResultMessage.Rows execute(){code}
because otherwise
{code:java}
public ResultMessage.Rows executeInternal() {code}
would stay without guard check or this guard call would have be duplicated.
Now, after reading the code, I understand that all Select statements will be
handled by that guard. Please check what you think about my change, if it is
"lets say ok" I will write tests :)
Thanks !
> Guardrail for page size
> ---
>
> Key: CASSANDRA-17189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17189
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Guardrails
>Reporter: Andres de la Peña
>Assignee: Bartlomiej
>Priority: Normal
> Labels: AdventCalendar2021, lhf
> Fix For: 4.1
>
> Attachments: CASSANDRA-17189-trunk.diff
>
>
> Add guardrail limiting the query page size, for example:
> {code}
> # Guardrail to warn about or reject page sizes greater than threshold.
> # The two thresholds default to -1 to disable.
> page_size:
> warn_threshold: -1
> abort_threshold: -1
> {code}
> Initially this can be based on the specified number of rows used as page
> size, although it would be ideal to also limit the actual size in bytes of
> the returned pages.
> +Additional information for newcomers:+
> # Add the configuration for the new guardrail on page size in the guardrails
> section of cassandra.yaml.
> # Add a getPageSize method in GuardrailsConfig returning a Threshold.Config
> object
> # Implement that method in GuardrailsOptions, which is the default yaml-based
> implementation of GuardrailsConfig
> # Add a Threshold guardrail named pageSize in Guardrails, using the
> previously created config
> # Define JMX-friendly getters and setters for the previously created config
> in GuardrailsMBean
> # Implement the JMX-friendly getters and setters in Guardrails
> # Now that we have the guardrail ready, it’s time to use it. We should search
> for a place to invoke the Guardrails.pageSize#guard method with the page size
> that each query is going to use. The DataLimits#forPaging methods look like
> good candidates for this.
> # Finally, add some tests for the new guardrail. Given that the new guardrail
> is a Threshold, our new test should probably extend ThresholdTester.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: com
[jira] [Comment Edited] (CASSANDRA-17189) Guardrail for page size
[
https://issues.apache.org/jira/browse/CASSANDRA-17189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17457359#comment-17457359
]
Bartlomiej edited comment on CASSANDRA-17189 at 12/10/21, 9:06 PM:
---
{code:java}
Why would it crash?{code}
that was because, in forPaging method, state was null
{code:java}
public DataLimits forPaging(int pageSize)
{
Guardrails.pageSize.guard(pageSize, "?", null);
return new CQLLimits(pageSize, perPartitionLimit, isDistinct);
}{code}
so guard was also applied for super user
{code:java}
(state == null || state.isOrdinaryUser()); {code}
Now it is not the case anymore because in execute I am able to get state and
super users are not aborded :)
{code:java}
By the way, that SelectStatement#columnFamily method uses the old terminology
for tables, which were originally called column families. Maybe we can rename
the method to table(). {code}
as you suggested I changed two methods to table() :)
I also had to move guard call from
{code:java}
public ResultMessage.Rows execute() {code}
to
{code:java}
private ResultMessage.Rows execute(){code}
because otherwise
{code:java}
public ResultMessage.Rows executeInternal() {code}
would stay without guard check or this guard call would have be duplicated.
Now, after reading the code, I understand that all Select statements will be
handled by that guard. Please check what you think about my change, if it is
"lets say ok" I will write tests :)
Thanks !
was (Author: bkowalczyyk):
{code:java}
Why would it crash?{code}
that was because, in forPaging method, state was null
{code:java}
public DataLimits forPaging(int pageSize)
{
Guardrails.pageSize.guard(pageSize, "?", null);
return new CQLLimits(pageSize, perPartitionLimit, isDistinct);
}{code}
so guard was also applied for super user
{code:java}
(state == null || state.isOrdinaryUser()); {code}
Now it is not the case anymore because in execute I am able to get state and
super users are not aborded :)
{code:java}
By the way, that SelectStatement#columnFamily method uses the old terminology
for tables, which were originally called column families. Maybe we can rename
the method to table(). {code}
as you suggested I changed two method to table() :)
I also had to move guard call from
{code:java}
public ResultMessage.Rows execute() {code}
to
{code:java}
private ResultMessage.Rows execute(){code}
because otherwise
{code:java}
public ResultMessage.Rows executeInternal() {code}
would stay without guard check or this guard call would have be duplicated.
Now, after reading the code, I understand that all Select statements will be
handled by that guard. Please check what you think about my change, if it is
"lets say ok" I will write tests :)
Thanks !
> Guardrail for page size
> ---
>
> Key: CASSANDRA-17189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17189
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Guardrails
>Reporter: Andres de la Peña
>Assignee: Bartlomiej
>Priority: Normal
> Labels: AdventCalendar2021, lhf
> Fix For: 4.1
>
> Attachments: CASSANDRA-17189-trunk.diff
>
>
> Add guardrail limiting the query page size, for example:
> {code}
> # Guardrail to warn about or reject page sizes greater than threshold.
> # The two thresholds default to -1 to disable.
> page_size:
> warn_threshold: -1
> abort_threshold: -1
> {code}
> Initially this can be based on the specified number of rows used as page
> size, although it would be ideal to also limit the actual size in bytes of
> the returned pages.
> +Additional information for newcomers:+
> # Add the configuration for the new guardrail on page size in the guardrails
> section of cassandra.yaml.
> # Add a getPageSize method in GuardrailsConfig returning a Threshold.Config
> object
> # Implement that method in GuardrailsOptions, which is the default yaml-based
> implementation of GuardrailsConfig
> # Add a Threshold guardrail named pageSize in Guardrails, using the
> previously created config
> # Define JMX-friendly getters and setters for the previously created config
> in GuardrailsMBean
> # Implement the JMX-friendly getters and setters in Guardrails
> # Now that we have the guardrail ready, it’s time to use it. We should search
> for a place to invoke the Guardrails.pageSize#guard method with the page size
> that each query is going to use. The DataLimits#forPaging methods look like
> good candidates for this.
> # Finally, add some tests for the new guardrail. Given that the new guardrail
> is a Threshold, our new test should probably extend ThresholdTester.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: c
[jira] [Commented] (CASSANDRA-17189) Guardrail for page size
[
https://issues.apache.org/jira/browse/CASSANDRA-17189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17457359#comment-17457359
]
Bartlomiej commented on CASSANDRA-17189:
{code:java}
Why would it crash?{code}
that was because, in forPaging method, state was null
{code:java}
public DataLimits forPaging(int pageSize)
{
Guardrails.pageSize.guard(pageSize, "?", null);
return new CQLLimits(pageSize, perPartitionLimit, isDistinct);
}{code}
so guard was also applied for super user
{code:java}
(state == null || state.isOrdinaryUser()); {code}
Now it is not the case anymore because in execute I am able to get state and
super users are not aborded :)
{code:java}
By the way, that SelectStatement#columnFamily method uses the old terminology
for tables, which were originally called column families. Maybe we can rename
the method to table(). {code}
as you suggested I changed two method to table() :)
I also had to move guard call from
{code:java}
public ResultMessage.Rows execute() {code}
to
{code:java}
private ResultMessage.Rows execute(){code}
because otherwise
{code:java}
public ResultMessage.Rows executeInternal() {code}
would stay without guard check or this guard call would have be duplicated.
Now, after reading the code, I understand that all Select statements will be
handled by that guard. Please check what you think about my change, if it is
"lets say ok" I will write tests :)
Thanks !
> Guardrail for page size
> ---
>
> Key: CASSANDRA-17189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17189
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Guardrails
>Reporter: Andres de la Peña
>Assignee: Bartlomiej
>Priority: Normal
> Labels: AdventCalendar2021, lhf
> Fix For: 4.1
>
> Attachments: CASSANDRA-17189-trunk.diff
>
>
> Add guardrail limiting the query page size, for example:
> {code}
> # Guardrail to warn about or reject page sizes greater than threshold.
> # The two thresholds default to -1 to disable.
> page_size:
> warn_threshold: -1
> abort_threshold: -1
> {code}
> Initially this can be based on the specified number of rows used as page
> size, although it would be ideal to also limit the actual size in bytes of
> the returned pages.
> +Additional information for newcomers:+
> # Add the configuration for the new guardrail on page size in the guardrails
> section of cassandra.yaml.
> # Add a getPageSize method in GuardrailsConfig returning a Threshold.Config
> object
> # Implement that method in GuardrailsOptions, which is the default yaml-based
> implementation of GuardrailsConfig
> # Add a Threshold guardrail named pageSize in Guardrails, using the
> previously created config
> # Define JMX-friendly getters and setters for the previously created config
> in GuardrailsMBean
> # Implement the JMX-friendly getters and setters in Guardrails
> # Now that we have the guardrail ready, it’s time to use it. We should search
> for a place to invoke the Guardrails.pageSize#guard method with the page size
> that each query is going to use. The DataLimits#forPaging methods look like
> good candidates for this.
> # Finally, add some tests for the new guardrail. Given that the new guardrail
> is a Threshold, our new test should probably extend ThresholdTester.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-17189) Guardrail for page size
[
https://issues.apache.org/jira/browse/CASSANDRA-17189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bartlomiej updated CASSANDRA-17189:
---
Attachment: CASSANDRA-17189-trunk.diff
> Guardrail for page size
> ---
>
> Key: CASSANDRA-17189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17189
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Guardrails
>Reporter: Andres de la Peña
>Assignee: Bartlomiej
>Priority: Normal
> Labels: AdventCalendar2021, lhf
> Fix For: 4.1
>
> Attachments: CASSANDRA-17189-trunk.diff
>
>
> Add guardrail limiting the query page size, for example:
> {code}
> # Guardrail to warn about or reject page sizes greater than threshold.
> # The two thresholds default to -1 to disable.
> page_size:
> warn_threshold: -1
> abort_threshold: -1
> {code}
> Initially this can be based on the specified number of rows used as page
> size, although it would be ideal to also limit the actual size in bytes of
> the returned pages.
> +Additional information for newcomers:+
> # Add the configuration for the new guardrail on page size in the guardrails
> section of cassandra.yaml.
> # Add a getPageSize method in GuardrailsConfig returning a Threshold.Config
> object
> # Implement that method in GuardrailsOptions, which is the default yaml-based
> implementation of GuardrailsConfig
> # Add a Threshold guardrail named pageSize in Guardrails, using the
> previously created config
> # Define JMX-friendly getters and setters for the previously created config
> in GuardrailsMBean
> # Implement the JMX-friendly getters and setters in Guardrails
> # Now that we have the guardrail ready, it’s time to use it. We should search
> for a place to invoke the Guardrails.pageSize#guard method with the page size
> that each query is going to use. The DataLimits#forPaging methods look like
> good candidates for this.
> # Finally, add some tests for the new guardrail. Given that the new guardrail
> is a Threshold, our new test should probably extend ThresholdTester.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-17189) Guardrail for page size
[
https://issues.apache.org/jira/browse/CASSANDRA-17189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bartlomiej updated CASSANDRA-17189:
---
Attachment: (was: CASSANDRA-17189-trunk.diff)
> Guardrail for page size
> ---
>
> Key: CASSANDRA-17189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17189
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Guardrails
>Reporter: Andres de la Peña
>Assignee: Bartlomiej
>Priority: Normal
> Labels: AdventCalendar2021, lhf
> Fix For: 4.1
>
>
> Add guardrail limiting the query page size, for example:
> {code}
> # Guardrail to warn about or reject page sizes greater than threshold.
> # The two thresholds default to -1 to disable.
> page_size:
> warn_threshold: -1
> abort_threshold: -1
> {code}
> Initially this can be based on the specified number of rows used as page
> size, although it would be ideal to also limit the actual size in bytes of
> the returned pages.
> +Additional information for newcomers:+
> # Add the configuration for the new guardrail on page size in the guardrails
> section of cassandra.yaml.
> # Add a getPageSize method in GuardrailsConfig returning a Threshold.Config
> object
> # Implement that method in GuardrailsOptions, which is the default yaml-based
> implementation of GuardrailsConfig
> # Add a Threshold guardrail named pageSize in Guardrails, using the
> previously created config
> # Define JMX-friendly getters and setters for the previously created config
> in GuardrailsMBean
> # Implement the JMX-friendly getters and setters in Guardrails
> # Now that we have the guardrail ready, it’s time to use it. We should search
> for a place to invoke the Guardrails.pageSize#guard method with the page size
> that each query is going to use. The DataLimits#forPaging methods look like
> good candidates for this.
> # Finally, add some tests for the new guardrail. Given that the new guardrail
> is a Threshold, our new test should probably extend ThresholdTester.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-17189) Guardrail for page size
[
https://issues.apache.org/jira/browse/CASSANDRA-17189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bartlomiej updated CASSANDRA-17189:
---
Attachment: CASSANDRA-17189-trunk.diff
> Guardrail for page size
> ---
>
> Key: CASSANDRA-17189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17189
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Guardrails
>Reporter: Andres de la Peña
>Assignee: Bartlomiej
>Priority: Normal
> Labels: AdventCalendar2021, lhf
> Fix For: 4.1
>
> Attachments: CASSANDRA-17189-trunk.diff
>
>
> Add guardrail limiting the query page size, for example:
> {code}
> # Guardrail to warn about or reject page sizes greater than threshold.
> # The two thresholds default to -1 to disable.
> page_size:
> warn_threshold: -1
> abort_threshold: -1
> {code}
> Initially this can be based on the specified number of rows used as page
> size, although it would be ideal to also limit the actual size in bytes of
> the returned pages.
> +Additional information for newcomers:+
> # Add the configuration for the new guardrail on page size in the guardrails
> section of cassandra.yaml.
> # Add a getPageSize method in GuardrailsConfig returning a Threshold.Config
> object
> # Implement that method in GuardrailsOptions, which is the default yaml-based
> implementation of GuardrailsConfig
> # Add a Threshold guardrail named pageSize in Guardrails, using the
> previously created config
> # Define JMX-friendly getters and setters for the previously created config
> in GuardrailsMBean
> # Implement the JMX-friendly getters and setters in Guardrails
> # Now that we have the guardrail ready, it’s time to use it. We should search
> for a place to invoke the Guardrails.pageSize#guard method with the page size
> that each query is going to use. The DataLimits#forPaging methods look like
> good candidates for this.
> # Finally, add some tests for the new guardrail. Given that the new guardrail
> is a Threshold, our new test should probably extend ThresholdTester.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-17189) Guardrail for page size
[
https://issues.apache.org/jira/browse/CASSANDRA-17189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bartlomiej updated CASSANDRA-17189:
---
Attachment: (was: CASSANDRA-17189-trunk.txt)
> Guardrail for page size
> ---
>
> Key: CASSANDRA-17189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17189
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Guardrails
>Reporter: Andres de la Peña
>Assignee: Bartlomiej
>Priority: Normal
> Labels: AdventCalendar2021, lhf
> Fix For: 4.1
>
>
> Add guardrail limiting the query page size, for example:
> {code}
> # Guardrail to warn about or reject page sizes greater than threshold.
> # The two thresholds default to -1 to disable.
> page_size:
> warn_threshold: -1
> abort_threshold: -1
> {code}
> Initially this can be based on the specified number of rows used as page
> size, although it would be ideal to also limit the actual size in bytes of
> the returned pages.
> +Additional information for newcomers:+
> # Add the configuration for the new guardrail on page size in the guardrails
> section of cassandra.yaml.
> # Add a getPageSize method in GuardrailsConfig returning a Threshold.Config
> object
> # Implement that method in GuardrailsOptions, which is the default yaml-based
> implementation of GuardrailsConfig
> # Add a Threshold guardrail named pageSize in Guardrails, using the
> previously created config
> # Define JMX-friendly getters and setters for the previously created config
> in GuardrailsMBean
> # Implement the JMX-friendly getters and setters in Guardrails
> # Now that we have the guardrail ready, it’s time to use it. We should search
> for a place to invoke the Guardrails.pageSize#guard method with the page size
> that each query is going to use. The DataLimits#forPaging methods look like
> good candidates for this.
> # Finally, add some tests for the new guardrail. Given that the new guardrail
> is a Threshold, our new test should probably extend ThresholdTester.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17189) Guardrail for page size
[
https://issues.apache.org/jira/browse/CASSANDRA-17189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17456819#comment-17456819
]
Bartlomiej commented on CASSANDRA-17189:
Hi [~brandon.williams] I will have a question, I made the change following
steps from the ticket [^CASSANDRA-17189-trunk.txt] , but I don't know what to
do here:
{code:java}
public DataLimits forPaging(int pageSize)
{
Guardrails.pageSize.guard(pageSize, "?", null);
return new CQLLimits(pageSize, perPartitionLimit, isDistinct);
}
public DataLimits forPaging(int pageSize, ByteBuffer lastReturnedKey, int
lastReturnedKeyRemaining)
{
Guardrails.pageSize.guard(pageSize, "?", null);
return new CQLPagingLimits(pageSize, perPartitionLimit, isDistinct,
lastReturnedKey, lastReturnedKeyRemaining);
} {code}
atm that is how messages look
{code:java}
WARN Quering page size with size 2200 exceeds warning threshold of 2000.
WARN Quering page size with size 2400 exceeds warning threshold of 2000. {code}
problem is that there is no table name in the log(what makes it difficult to
figure out what query exceed the threshold), also guard requires 'what'
argument (it is is place for table name), but I have no idea how to make it
good :) should I pass table name to forPaging ? that doesn't looks good for me
(or no?). Should I extract guard outside forPaging ?(for example in
SelectStatement.execute?) - problem is that there are multiple places that
forPagining is executed.
* I also wonder, should we care about too small value for abort_threshold ? If
someone will set for example 100, cassandra will crash.
Thanks !
> Guardrail for page size
> ---
>
> Key: CASSANDRA-17189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17189
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Guardrails
>Reporter: Andres de la Peña
>Assignee: Bartlomiej
>Priority: Normal
> Labels: AdventCalendar2021, lhf
> Fix For: 4.1
>
> Attachments: CASSANDRA-17189-trunk.txt
>
>
> Add guardrail limiting the query page size, for example:
> {code}
> # Guardrail to warn about or reject page sizes greater than threshold.
> # The two thresholds default to -1 to disable.
> page_size:
> warn_threshold: -1
> abort_threshold: -1
> {code}
> Initially this can be based on the specified number of rows used as page
> size, although it would be ideal to also limit the actual size in bytes of
> the returned pages.
> +Additional information for newcomers:+
> # Add the configuration for the new guardrail on page size in the guardrails
> section of cassandra.yaml.
> # Add a getPageSize method in GuardrailsConfig returning a Threshold.Config
> object
> # Implement that method in GuardrailsOptions, which is the default yaml-based
> implementation of GuardrailsConfig
> # Add a Threshold guardrail named pageSize in Guardrails, using the
> previously created config
> # Define JMX-friendly getters and setters for the previously created config
> in GuardrailsMBean
> # Implement the JMX-friendly getters and setters in Guardrails
> # Now that we have the guardrail ready, it’s time to use it. We should search
> for a place to invoke the Guardrails.pageSize#guard method with the page size
> that each query is going to use. The DataLimits#forPaging methods look like
> good candidates for this.
> # Finally, add some tests for the new guardrail. Given that the new guardrail
> is a Threshold, our new test should probably extend ThresholdTester.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-17189) Guardrail for page size
[
https://issues.apache.org/jira/browse/CASSANDRA-17189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17456819#comment-17456819
]
Bartlomiej edited comment on CASSANDRA-17189 at 12/9/21, 11:33 PM:
---
Hi [~brandon.williams] I will have a question, I made the change following
steps from the ticket [^CASSANDRA-17189-trunk.txt] , but I don't know what to
do here:
{code:java}
public DataLimits forPaging(int pageSize)
{
Guardrails.pageSize.guard(pageSize, "?", null);
return new CQLLimits(pageSize, perPartitionLimit, isDistinct);
}
public DataLimits forPaging(int pageSize, ByteBuffer lastReturnedKey, int
lastReturnedKeyRemaining)
{
Guardrails.pageSize.guard(pageSize, "?", null);
return new CQLPagingLimits(pageSize, perPartitionLimit, isDistinct,
lastReturnedKey, lastReturnedKeyRemaining);
} {code}
atm that is how messages look
{code:java}
WARN Quering page size with size 2200 exceeds warning threshold of 2000.
WARN Quering page size with size 2400 exceeds warning threshold of 2000. {code}
problem is that there is no table name in the log(what makes it difficult to
figure out what query exceed the threshold), also guard requires 'what'
argument (it is is place for table name), but I have no idea how to make it
good :) should I pass table name to forPaging ? that doesn't looks good for me
(or no?). Should I extract guard outside forPaging ?(for example in
SelectStatement.execute?) - problem is that there are multiple places that
forPagining is executed.
I also wonder, should we care about too small value for abort_threshold ? If
someone will set for example 100, cassandra will crash.
Thanks !
was (Author: bkowalczyyk):
Hi [~brandon.williams] I will have a question, I made the change following
steps from the ticket [^CASSANDRA-17189-trunk.txt] , but I don't know what to
do here:
{code:java}
public DataLimits forPaging(int pageSize)
{
Guardrails.pageSize.guard(pageSize, "?", null);
return new CQLLimits(pageSize, perPartitionLimit, isDistinct);
}
public DataLimits forPaging(int pageSize, ByteBuffer lastReturnedKey, int
lastReturnedKeyRemaining)
{
Guardrails.pageSize.guard(pageSize, "?", null);
return new CQLPagingLimits(pageSize, perPartitionLimit, isDistinct,
lastReturnedKey, lastReturnedKeyRemaining);
} {code}
atm that is how messages look
{code:java}
WARN Quering page size with size 2200 exceeds warning threshold of 2000.
WARN Quering page size with size 2400 exceeds warning threshold of 2000. {code}
problem is that there is no table name in the log(what makes it difficult to
figure out what query exceed the threshold), also guard requires 'what'
argument (it is is place for table name), but I have no idea how to make it
good :) should I pass table name to forPaging ? that doesn't looks good for me
(or no?). Should I extract guard outside forPaging ?(for example in
SelectStatement.execute?) - problem is that there are multiple places that
forPagining is executed.
* I also wonder, should we care about too small value for abort_threshold ? If
someone will set for example 100, cassandra will crash.
Thanks !
> Guardrail for page size
> ---
>
> Key: CASSANDRA-17189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17189
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Guardrails
>Reporter: Andres de la Peña
>Assignee: Bartlomiej
>Priority: Normal
> Labels: AdventCalendar2021, lhf
> Fix For: 4.1
>
> Attachments: CASSANDRA-17189-trunk.txt
>
>
> Add guardrail limiting the query page size, for example:
> {code}
> # Guardrail to warn about or reject page sizes greater than threshold.
> # The two thresholds default to -1 to disable.
> page_size:
> warn_threshold: -1
> abort_threshold: -1
> {code}
> Initially this can be based on the specified number of rows used as page
> size, although it would be ideal to also limit the actual size in bytes of
> the returned pages.
> +Additional information for newcomers:+
> # Add the configuration for the new guardrail on page size in the guardrails
> section of cassandra.yaml.
> # Add a getPageSize method in GuardrailsConfig returning a Threshold.Config
> object
> # Implement that method in GuardrailsOptions, which is the default yaml-based
> implementation of GuardrailsConfig
> # Add a Threshold guardrail named pageSize in Guardrails, using the
> previously created config
> # Define JMX-friendly getters and setters for the previously created config
> in GuardrailsMBean
> # Implement the JMX-friendly getters and setters in Guardrails
> # Now that we have the guardrail ready, it’s time to use it. We should search
> for a place to invoke the Guardrails.pageSize#guard method with the page size
> that each query is going to
[jira] [Updated] (CASSANDRA-17189) Guardrail for page size
[
https://issues.apache.org/jira/browse/CASSANDRA-17189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bartlomiej updated CASSANDRA-17189:
---
Attachment: CASSANDRA-17189-trunk.txt
> Guardrail for page size
> ---
>
> Key: CASSANDRA-17189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17189
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Guardrails
>Reporter: Andres de la Peña
>Assignee: Bartlomiej
>Priority: Normal
> Labels: AdventCalendar2021, lhf
> Fix For: 4.1
>
> Attachments: CASSANDRA-17189-trunk.txt
>
>
> Add guardrail limiting the query page size, for example:
> {code}
> # Guardrail to warn about or reject page sizes greater than threshold.
> # The two thresholds default to -1 to disable.
> page_size:
> warn_threshold: -1
> abort_threshold: -1
> {code}
> Initially this can be based on the specified number of rows used as page
> size, although it would be ideal to also limit the actual size in bytes of
> the returned pages.
> +Additional information for newcomers:+
> # Add the configuration for the new guardrail on page size in the guardrails
> section of cassandra.yaml.
> # Add a getPageSize method in GuardrailsConfig returning a Threshold.Config
> object
> # Implement that method in GuardrailsOptions, which is the default yaml-based
> implementation of GuardrailsConfig
> # Add a Threshold guardrail named pageSize in Guardrails, using the
> previously created config
> # Define JMX-friendly getters and setters for the previously created config
> in GuardrailsMBean
> # Implement the JMX-friendly getters and setters in Guardrails
> # Now that we have the guardrail ready, it’s time to use it. We should search
> for a place to invoke the Guardrails.pageSize#guard method with the page size
> that each query is going to use. The DataLimits#forPaging methods look like
> good candidates for this.
> # Finally, add some tests for the new guardrail. Given that the new guardrail
> is a Threshold, our new test should probably extend ThresholdTester.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17189) Guardrail for page size
[
https://issues.apache.org/jira/browse/CASSANDRA-17189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17455913#comment-17455913
]
Bartlomiej commented on CASSANDRA-17189:
Hi,
I would like to try implement this (hope it will not overwhelm me :D ),
thanks !
> Guardrail for page size
> ---
>
> Key: CASSANDRA-17189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17189
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Guardrails
>Reporter: Andres de la Peña
>Priority: Normal
> Labels: AdventCalendar2021, lhf
> Fix For: 4.1
>
>
> Add guardrail limiting the query page size, for example:
> {code}
> # Guardrail to warn about or reject page sizes greater than threshold.
> # The two thresholds default to -1 to disable.
> page_size:
> warn_threshold: -1
> abort_threshold: -1
> {code}
> Initially this can be based on the specified number of rows used as page
> size, although it would be ideal to also limit the actual size in bytes of
> the returned pages.
> +Additional information for newcomers:+
> # Add the configuration for the new guardrail on page size in the guardrails
> section of cassandra.yaml.
> # Add a getPageSize method in GuardrailsConfig returning a Threshold.Config
> object
> # Implement that method in GuardrailsOptions, which is the default yaml-based
> implementation of GuardrailsConfig
> # Add a Threshold guardrail named pageSize in Guardrails, using the
> previously created config
> # Define JMX-friendly getters and setters for the previously created config
> in GuardrailsMBean
> # Implement the JMX-friendly getters and setters in Guardrails
> # Now that we have the guardrail ready, it’s time to use it. We should search
> for a place to invoke the Guardrails.pageSize#guard method with the page size
> that each query is going to use. The DataLimits#forPaging methods look like
> good candidates for this.
> # Finally, add some tests for the new guardrail. Given that the new guardrail
> is a Threshold, our new test should probably extend ThresholdTester.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17181) SchemaCQLHelperTest methods can be simplified
[
https://issues.apache.org/jira/browse/CASSANDRA-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17454712#comment-17454712
]
Bartlomiej commented on CASSANDRA-17181:
Hi,
I created patch following steps from
[https://cassandra.apache.org/_/development/patches.html]
I looked if I can remove properties you pointed, so it is a bit simpler now. I
run tests I modified (all passed). I wondered about TableMatadata.toCqlString
because it also is used once with (true, true true) passed, but I was not sure
I should touch it :)
I also run the *ant test* but I took almost 3h to finish and there was some
timeout :) so I am not sure about testing phase of this change, any
recommendations ?
thanks !
> SchemaCQLHelperTest methods can be simplified
> -
>
> Key: CASSANDRA-17181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17181
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: Benjamin Lerer
>Assignee: Bartlomiej
>Priority: Normal
> Labels: AdventCalendar2021, lhf
> Fix For: 4.1
>
> Attachments: CASSANDRA-17181-4.0.patch
>
>
> {{SchemaCQLHelperTest}} is used during a snapshot to generate the
> {{schema.cql}} file. The methods accept the following paramaters:
> {{includeDroppedColumns}}, {{internals}} and {{ifNotExists}}.
> Those parameters are in practice always set to true by the calling code and
> therefore can be removed.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-17181) SchemaCQLHelperTest methods can be simplified
[
https://issues.apache.org/jira/browse/CASSANDRA-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bartlomiej updated CASSANDRA-17181:
---
Attachment: CASSANDRA-17181-4.0.patch
> SchemaCQLHelperTest methods can be simplified
> -
>
> Key: CASSANDRA-17181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17181
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: Benjamin Lerer
>Assignee: Bartlomiej
>Priority: Normal
> Labels: AdventCalendar2021, lhf
> Fix For: 4.1
>
> Attachments: CASSANDRA-17181-4.0.patch
>
>
> {{SchemaCQLHelperTest}} is used during a snapshot to generate the
> {{schema.cql}} file. The methods accept the following paramaters:
> {{includeDroppedColumns}}, {{internals}} and {{ifNotExists}}.
> Those parameters are in practice always set to true by the calling code and
> therefore can be removed.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17181) schema.cql should have IF NOT EXISTS for CREATE TYPE
[
https://issues.apache.org/jira/browse/CASSANDRA-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17453949#comment-17453949
]
Bartlomiej commented on CASSANDRA-17181:
{code:java}
If you are interested in cleaning that I can update the ticket to reflect the
new goal {code}
yeas pls - I will clean this :)
> schema.cql should have IF NOT EXISTS for CREATE TYPE
>
>
> Key: CASSANDRA-17181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17181
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: Benjamin Lerer
>Priority: Normal
> Labels: AdventCalendar2021, lhf
> Fix For: 3.0.x, 3.11.x, 4.0.x
>
>
> When a schema.cql is created during a snapshot, {{CREATE TYPE}} statements
> are not created with {{IF NOT EXISTS}} as the {{CREATE TABLE}} statements.
> This can cause some issues when using those statements during restore.
> Additional Information for newcomers:
> In 3.0 and 3.11 the code use to generate the {{schema.cql}} file is in
> {{TableCQLHelper}} and the unit tests in {{TableCQLHelperTest}}.
> In 4.0 the bug does not exist anymore but the test {{SchemaCQLHelperTest}}
> does not test the correct behavior and the code could be simplified
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-17181) schema.cql should have IF NOT EXISTS for CREATE TYPE
[
https://issues.apache.org/jira/browse/CASSANDRA-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17453666#comment-17453666
]
Bartlomiej edited comment on CASSANDRA-17181 at 12/5/21, 8:01 PM:
--
Hi [~blerer], I wanted to work on that, so I looked at the source code first, I
am not sure if this
{code:java}
In 3.0 and 3.11 the code use to generate the schema.cql file is in
TableCQLHelper and the unit tests in TableCQLHelperTest {code}
is true. In *cassandra-3.11* (also true in {*}cassandra-3.0{*}) branch I see
that there is no TableCQLHelper class and instead of TableCQLHelperTest,
ColumnFamilyStoreCQLHelperTest exists, which (as far I understand) tests
snapshot creation and also handle the case described in the ticket
{code:java}
String schema =
Files.toString(cfs.getDirectories().getSnapshotSchemaFile(SNAPSHOT),
Charset.defaultCharset());
assertTrue(schema.contains(String.format("CREATE TYPE IF NOT EXISTS %s.%s(a1
varint, a2 varint, a3 varint);", keyspace(), typeA)));
assertTrue(schema.contains(String.format("CREATE TYPE IF NOT EXISTS %s.%s(a1
varint, a2 varint, a3 varint);", keyspace(), typeA)));
{code}
can you hint me more about this issue - I am wrong or right ?
thanks !
was (Author: bkowalczyyk):
Hi [~blerer], I wanted to work on that, so I looked at the source code first, I
am not sure if this
{code:java}
In 3.0 and 3.11 the code use to generate the schema.cql file is in
TableCQLHelper and the unit tests in TableCQLHelperTest {code}
is true. In *cassandra-3.11* (also true in {*}cassandra-3.0{*}) branch I see
that there is no TableCQLHelper class and instead of TableCQLHelperTest,
ColumnFamilyStoreCQLHelperTest exists, which (as far I understand) tests
snapshot creation and also handle the case described in the ticket
{code:java}
String schema =
Files.toString(cfs.getDirectories().getSnapshotSchemaFile(SNAPSHOT),
Charset.defaultCharset());
assertTrue(schema.contains(String.format("CREATE TYPE IF NOT EXISTS %s.%s(a1
varint, a2 varint, a3 varint);", keyspace(), typeA)));
assertTrue(schema.contains(String.format("CREATE TYPE IF NOT EXISTS %s.%s(a1
varint, a2 varint, a3 varint);", keyspace(), typeA)));
{code}
can you hint me more about this issue - I am wrong or right ?
> schema.cql should have IF NOT EXISTS for CREATE TYPE
>
>
> Key: CASSANDRA-17181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17181
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: Benjamin Lerer
>Priority: Normal
> Labels: AdventCalendar2021, lhf
> Fix For: 3.0.x, 3.11.x, 4.0.x
>
>
> When a schema.cql is created during a snapshot, {{CREATE TYPE}} statements
> are not created with {{IF NOT EXISTS}} as the {{CREATE TABLE}} statements.
> This can cause some issues when using those statements during restore.
> Additional Information for newcomers:
> In 3.0 and 3.11 the code use to generate the {{schema.cql}} file is in
> {{TableCQLHelper}} and the unit tests in {{TableCQLHelperTest}}.
> In 4.0 the bug does not exist anymore but the test {{SchemaCQLHelperTest}}
> does not test the correct behavior and the code could be simplified
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17181) schema.cql should have IF NOT EXISTS for CREATE TYPE
[
https://issues.apache.org/jira/browse/CASSANDRA-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17453666#comment-17453666
]
Bartlomiej commented on CASSANDRA-17181:
Hi [~blerer], I wanted to work on that, so I looked at the source code first, I
am not sure if this
{code:java}
In 3.0 and 3.11 the code use to generate the schema.cql file is in
TableCQLHelper and the unit tests in TableCQLHelperTest {code}
is true. In *cassandra-3.11* (also true in {*}cassandra-3.0{*}) branch I see
that there is no TableCQLHelper class and instead of TableCQLHelperTest,
ColumnFamilyStoreCQLHelperTest exists, which (as far I understand) tests
snapshot creation and also handle the case described in the ticket
{code:java}
String schema =
Files.toString(cfs.getDirectories().getSnapshotSchemaFile(SNAPSHOT),
Charset.defaultCharset());
assertTrue(schema.contains(String.format("CREATE TYPE IF NOT EXISTS %s.%s(a1
varint, a2 varint, a3 varint);", keyspace(), typeA)));
assertTrue(schema.contains(String.format("CREATE TYPE IF NOT EXISTS %s.%s(a1
varint, a2 varint, a3 varint);", keyspace(), typeA)));
{code}
can you hint me more about this issue - I am wrong or right ?
> schema.cql should have IF NOT EXISTS for CREATE TYPE
>
>
> Key: CASSANDRA-17181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17181
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: Benjamin Lerer
>Priority: Normal
> Labels: AdventCalendar2021, lhf
> Fix For: 3.0.x, 3.11.x, 4.0.x
>
>
> When a schema.cql is created during a snapshot, {{CREATE TYPE}} statements
> are not created with {{IF NOT EXISTS}} as the {{CREATE TABLE}} statements.
> This can cause some issues when using those statements during restore.
> Additional Information for newcomers:
> In 3.0 and 3.11 the code use to generate the {{schema.cql}} file is in
> {{TableCQLHelper}} and the unit tests in {{TableCQLHelperTest}}.
> In 4.0 the bug does not exist anymore but the test {{SchemaCQLHelperTest}}
> does not test the correct behavior and the code could be simplified
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11988) NullPointerExpception when reading/compacting table
[
https://issues.apache.org/jira/browse/CASSANDRA-11988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334641#comment-15334641
]
Bartlomiej commented on CASSANDRA-11988:
[~carlyeks] according to "I'm worried that other places might have the same
issue;". Can deletion of those static columns protect us from such data
corruption in future ?
> NullPointerExpception when reading/compacting table
> ---
>
> Key: CASSANDRA-11988
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11988
> Project: Cassandra
> Issue Type: Bug
>Reporter: Nimi Wariboko Jr.
>Assignee: Carl Yeksigian
> Fix For: 3.6
>
>
> I have a table that suddenly refuses to be read or compacted. Issuing a read
> on the table causes a NPE.
> On compaction, it returns the error
> {code}
> ERROR [CompactionExecutor:6] 2016-06-09 17:10:15,724 CassandraDaemon.java:213
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:38)
> ~[apache-cassandra-3.6.jar:3.6]
> at
> org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:64)
> ~[apache-cassandra-3.6.jar:3.6]
> at
> org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24)
> ~[apache-cassandra-3.6.jar:3.6]
> at
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:76)
> ~[apache-cassandra-3.6.jar:3.6]
> at
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
> ~[apache-cassandra-3.6.jar:3.6]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182)
> ~[apache-cassandra-3.6.jar:3.6]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.6.jar:3.6]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:82)
> ~[apache-cassandra-3.6.jar:3.6]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
> ~[apache-cassandra-3.6.jar:3.6]
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
> ~[apache-cassandra-3.6.jar:3.6]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> {code}
> Schema:
> {code}
> CREATE TABLE cmpayments.report_payments (
> reportid timeuuid,
> userid timeuuid,
> adjustedearnings decimal,
> deleted set static,
> earnings map,
> gross map,
> organizationid text,
> payall timestamp static,
> status text,
> PRIMARY KEY (reportid, userid)
> ) WITH CLUSTERING ORDER BY (userid ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11988) NullPointerExpception when reading/compacting table
[
https://issues.apache.org/jira/browse/CASSANDRA-11988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329066#comment-15329066
]
Bartlomiej commented on CASSANDRA-11988:
Same issue with Cassandra 3.0.6
> NullPointerExpception when reading/compacting table
> ---
>
> Key: CASSANDRA-11988
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11988
> Project: Cassandra
> Issue Type: Bug
>Reporter: Nimi Wariboko Jr.
> Fix For: 3.6
>
>
> I have a table that suddenly refuses to be read or compacted. Issuing a read
> on the table causes a NPE.
> On compaction, it returns the error
> {code}
> ERROR [CompactionExecutor:6] 2016-06-09 17:10:15,724 CassandraDaemon.java:213
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:38)
> ~[apache-cassandra-3.6.jar:3.6]
> at
> org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:64)
> ~[apache-cassandra-3.6.jar:3.6]
> at
> org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24)
> ~[apache-cassandra-3.6.jar:3.6]
> at
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:76)
> ~[apache-cassandra-3.6.jar:3.6]
> at
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
> ~[apache-cassandra-3.6.jar:3.6]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182)
> ~[apache-cassandra-3.6.jar:3.6]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.6.jar:3.6]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:82)
> ~[apache-cassandra-3.6.jar:3.6]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
> ~[apache-cassandra-3.6.jar:3.6]
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
> ~[apache-cassandra-3.6.jar:3.6]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> {code}
> Schema:
> {code}
> CREATE TABLE cmpayments.report_payments (
> reportid timeuuid,
> userid timeuuid,
> adjustedearnings decimal,
> deleted set static,
> earnings map,
> gross map,
> organizationid text,
> payall timestamp static,
> status text,
> PRIMARY KEY (reportid, userid)
> ) WITH CLUSTERING ORDER BY (userid ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
