[jira] [Comment Edited] (CASSANDRA-16104) Wrong warning about data volumes capacity
[
https://issues.apache.org/jira/browse/CASSANDRA-16104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17386199#comment-17386199
]
Andres de la Peña edited comment on CASSANDRA-16104 at 7/23/21, 12:14 PM:
--
As suggested by [~Bereng] and [~djatnieks], from 4.0 we can have [a JUnit
test|https://github.com/adelapena/cassandra/blob/38c1a41ddeb1b75f9c57a3ec6900d8638c01ddb2/test/unit/org/apache/cassandra/tools/NodeProbeTest.java]
instead of the Python dtest. I have added it to the patches, rebased and rerun
CI:
||patch||CI||
|[3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...adelapena:16104-3.0]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/691/workflows/4f824afb-9fdf-438f-b50d-6c571fde297b]|
|[3.11|https://github.com/apache/cassandra/compare/cassandra-3.11...adelapena:16104-3.11]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/692/workflows/efc91371-1999-47f3-acb6-21a1024e1bd2]|
|[4.0|https://github.com/apache/cassandra/compare/cassandra-4.0...adelapena:16104-4.0]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/696/workflows/76a55c1c-ef4f-4c42-9b31-f40f934604b5]
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/696/workflows/d32878f3-f1ee-4604-a70a-d0a82790c684]|
|[trunk|https://github.com/apache/cassandra/compare/trunk...adelapena:16104-trunk]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/695/workflows/9244b3d6-a31c-4e6c-b2c0-39761b235885]
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/695/workflows/d5b504c8-b61b-47c2-974d-70e938ddcb51]|
|[dtest|https://github.com/apache/cassandra-dtest/compare/trunk...adelapena:CASSANDRA-16104]||
If that looks good we'll only need the approval of a second reviewer.
was (Author: adelapena):
As suggested by [~Bereng] and [~djatnieks], from 4.0 we can have [a JUnit
test|https://github.com/adelapena/cassandra/blob/38c1a41ddeb1b75f9c57a3ec6900d8638c01ddb2/test/unit/org/apache/cassandra/tools/NodeProbeTest.java]
instead of the Python dtest. I have added it to the patches, rebased and rerun
CI:
||patch||CI||
|[3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...adelapena:16104-3.0]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/691/workflows/4f824afb-9fdf-438f-b50d-6c571fde297b]|
|[3.11|https://github.com/apache/cassandra/compare/cassandra-3.11...adelapena:16104-3.11]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/692/workflows/efc91371-1999-47f3-acb6-21a1024e1bd2]|
|[4.0|https://github.com/apache/cassandra/compare/cassandra-4.0...adelapena:16104-4.0]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/693/workflows/66bbd6bc-1656-4c3f-9723-769842b875a0]
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/693/workflows/5bd5ca40-e019-472a-8278-b27a65f8cabe]|
|[trunk|https://github.com/apache/cassandra/compare/trunk...adelapena:16104-trunk]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/694/workflows/c390058b-8a7b-4e12-871a-41b26f19dfb1]
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/694/workflows/f55ab5a3-770b-46e8-9052-be00232884bb]|
|[dtest|https://github.com/apache/cassandra-dtest/compare/trunk...adelapena:CASSANDRA-16104]|
If that looks good we'll only need the approval of a second reviewer.
> Wrong warning about data volumes capacity
> -
>
> Key: CASSANDRA-16104
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16104
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Dmitrii Saprykin
>Assignee: Dmitrii Saprykin
>Priority: Normal
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I see the following warning trying to run *nodetool upgradesstables*
> {noformat}
> WARN 16:09:24 Only 34988 MB free across all data volumes. Consider adding
> more capacity to your cluster or removing obsolete snapshots
> {noformat}
> This warning is wrong because the wrong storage device capacity gets tested.
> All my cassandra data paths are subdirectories of */data* mount point which
> has enough of space:
> {noformat}
> $ df -h /data
> Filesystem Size Used Avail Mounted on
> . 1.2T 200G 1T /data
> {noformat}
> However what Warning reports is a OS mount which has nothing to do with
> Cassandra configuration:
> {noformat}
> df -h /
> Filesystem Size Used Avail Use% Mounted on
> 40G 5.7G 35G 15% /
> {noformat}
> I see this error running Cassandra 3.0.22
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits
[jira] [Comment Edited] (CASSANDRA-16104) Wrong warning about data volumes capacity
[
https://issues.apache.org/jira/browse/CASSANDRA-16104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17380576#comment-17380576
]
Andres de la Peña edited comment on CASSANDRA-16104 at 7/14/21, 1:05 PM:
-
[~saprykin] Thanks for the patch, and sorry for the delayed review.
Getting concurrent compactors through JMX when validating the {{-j/--jobs}}
option makes sense to me. Indeed, there is [a TODO
comment|https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/tools/NodeProbe.java#L302]
about this in 3.11, dating back to CASSANDRA-9054. I have rebased the patch
and created versions for 3.11, 4.0 and trunk, and CI is running:
||patch||CI||
|[3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...adelapena:16104-3.0]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/661/workflows/3e91ee7c-eb67-416b-af65-3e0314e5d359]|
|[3.11|https://github.com/apache/cassandra/compare/cassandra-3.11...adelapena:16104-3.11]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/664/workflows/2fe0f649-b846-47e4-8a42-74c04aef75c4]|
|[4.0|https://github.com/apache/cassandra/compare/cassandra-4.0...adelapena:16104-4.0]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/663/workflows/e58e0821-b6a7-4b18-923d-f0eddf743734]
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/663/workflows/be12b427-30ed-429b-9a4e-ca0ec1a5e53a]|
|[trunk|https://github.com/apache/cassandra/compare/trunk...adelapena:16104-trunk]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/662/workflows/bf7ad882-b6f7-4c13-a534-0e64b740c748]
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/662/workflows/b1ff8a2f-b268-407f-98ef-cb13b10fc575]|
>From 3.11 and beyond we don't need to add the mbean methods since they already
>exist since CASSANDRA-12248, so we only need to use them from {{NodeProbe}} to
>get rid of all the dependencies from {{DatabaseDescriptor}}.
I wonder whether it's worth adding a dtest setting a couple of nodes with
different values for {{concurrent_compactors}} and call them with nodetool to
verify that the {{-j/--jobs}} option issues the warning depending on the
connected node. I think it should be a Python dtest since, if I'm right, Java
dtests use an internal probe that doesn't use JMX.
was (Author: adelapena):
[~saprykin] Thanks for the patch, and sorry for the delayed review.
Getting concurrent compactors through JMX when validating the {{-j/--jobs}}
option makes sense to me. Indeed, there is [a TODO
comment|https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/tools/NodeProbe.java#L302]
about this in 3.11, dating back to CASSANDRA-9054. I have rebased the patch
and created versions for 3.11, 4.0 and trunk, and CI is running:
||patch||CI||
|[3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...adelapena:16104-3.0]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/661/workflows/3e91ee7c-eb67-416b-af65-3e0314e5d359]|
|[3.11|https://github.com/apache/cassandra/compare/cassandra-3.11...adelapena:16104-3.11]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/664/workflows/2fe0f649-b846-47e4-8a42-74c04aef75c4]|
|[4.0|https://github.com/apache/cassandra/compare/cassandra-4.0...adelapena:16104-4.0]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/663/workflows/e58e0821-b6a7-4b18-923d-f0eddf743734]
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/663/workflows/be12b427-30ed-429b-9a4e-ca0ec1a5e53a]|
|[trunk|https://github.com/apache/cassandra/compare/cassandra-trunk...adelapena:16104-trunk]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/662/workflows/bf7ad882-b6f7-4c13-a534-0e64b740c748]
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/662/workflows/b1ff8a2f-b268-407f-98ef-cb13b10fc575]|
>From 3.11 and beyond we don't need to add the mbean methods since they already
>exist since CASSANDRA-12248, so we only need to use them from {{NodeProbe}} to
>get rid of all the dependencies from {{DatabaseDescriptor}}.
I wonder whether it's worth adding a dtest setting a couple of nodes with
different values for {{concurrent_compactors}} and call them with nodetool to
verify that the {{-j/--jobs}} option issues the warning depending on the
connected node. I think it should be a Python dtest since, if I'm right, Java
dtests use an internal probe that doesn't use JMX.
> Wrong warning about data volumes capacity
> -
>
> Key: CASSANDRA-16104
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16104
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Dmitrii Saprykin
>Assignee: Dmitrii Saprykin
[jira] [Comment Edited] (CASSANDRA-16104) Wrong warning about data volumes capacity
[
https://issues.apache.org/jira/browse/CASSANDRA-16104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210330#comment-17210330
]
Dmitrii Saprykin edited comment on CASSANDRA-16104 at 10/8/20, 4:56 PM:
After some digging I found a source for this warning:
* src/java/org/apache/cassandra/tools/NodeProbe.java method *upgradeSSTables*
calls *checkJobs* to validate *--jobs* parameter ;
* which calls *DatabaseDescriptor.toolInitialization(false)* ;
* *DatabaseDescriptor* calls *applySimpleConfig* which validates
*data_files_directories* and produces warning ;
So data directory space gets checked in the wrong location (if CASSANDRA_CONF
env is not default) and on the wrong host (it is checked on nodetool
upgradesstables request source).
Moreover I do not think checkJobs has any sense at all as it validates *--jobs*
against local configuration which has nothing to do with
*concurrent_compactors* value for host which actually executes upgrade sstables
operation.
was (Author: saprykin):
After some digging I found a source for this warning:
* src/java/org/apache/cassandra/tools/NodeProbe.java method *upgradeSSTables*
calls *checkJobs* to validate *--jobs* parameter ;
* which calls *DatabaseDescriptor.toolInitialization(false)* ;
* *DatabaseDescriptor* calls *applySimpleConfig* which validates
*data_files_directories* and produces warning ;
So data directory space gets checked in the wrong location (if CASSANDRA_CONF
env is not default) and on the wrong host (it is checked on nodetool
upgradesstables request source).
> Wrong warning about data volumes capacity
> -
>
> Key: CASSANDRA-16104
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16104
> Project: Cassandra
> Issue Type: Bug
>Reporter: Dmitrii Saprykin
>Priority: Normal
>
> I see the following warning trying to run *nodetool upgradesstables*
> {noformat}
> WARN 16:09:24 Only 34988 MB free across all data volumes. Consider adding
> more capacity to your cluster or removing obsolete snapshots
> {noformat}
> This warning is wrong because the wrong storage device capacity gets tested.
> All my cassandra data paths are subdirectories of */data* mount point which
> has enough of space:
> {noformat}
> $ df -h /data
> Filesystem Size Used Avail Mounted on
> . 1.2T 200G 1T /data
> {noformat}
> However what Warning reports is a OS mount which has nothing to do with
> Cassandra configuration:
> {noformat}
> df -h /
> Filesystem Size Used Avail Use% Mounted on
> 40G 5.7G 35G 15% /
> {noformat}
> I see this error running Cassandra 3.0.22
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16104) Wrong warning about data volumes capacity
[
https://issues.apache.org/jira/browse/CASSANDRA-16104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210330#comment-17210330
]
Dmitrii Saprykin edited comment on CASSANDRA-16104 at 10/8/20, 4:52 PM:
After some digging I found a source for this warning:
* src/java/org/apache/cassandra/tools/NodeProbe.java method *upgradeSSTables*
calls *checkJobs* to validate *--jobs* parameter ;
* which calls *DatabaseDescriptor.toolInitialization(false)* ;
* *DatabaseDescriptor* calls *applySimpleConfig* which validates
*data_files_directories* and produces warning ;
So data directory space gets checked in the wrong location (if CASSANDRA_CONF
env is not default) and on the wrong host (it is checked on nodetool
upgradesstables request source).
was (Author: saprykin):
After some digging I found a source for this warning:
* src/java/org/apache/cassandra/tools/NodeProbe.java method upgradeSSTables
calls *checkJobs* to validate *--jobs* parameter ;
* which calls DatabaseDescriptor.toolInitialization(false) ;
* DatabaseDescriptor calls *applySimpleConfig* which validates
*data_files_directories* and produces warning ;
So data directory space gets checked in the wrong location (if CASSANDRA_CONF
env is not default) and on the wrong host (it is checked on nodetool
upgradesstables request source).
> Wrong warning about data volumes capacity
> -
>
> Key: CASSANDRA-16104
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16104
> Project: Cassandra
> Issue Type: Bug
>Reporter: Dmitrii Saprykin
>Priority: Normal
>
> I see the following warning trying to run *nodetool upgradesstables*
> {noformat}
> WARN 16:09:24 Only 34988 MB free across all data volumes. Consider adding
> more capacity to your cluster or removing obsolete snapshots
> {noformat}
> This warning is wrong because the wrong storage device capacity gets tested.
> All my cassandra data paths are subdirectories of */data* mount point which
> has enough of space:
> {noformat}
> $ df -h /data
> Filesystem Size Used Avail Mounted on
> . 1.2T 200G 1T /data
> {noformat}
> However what Warning reports is a OS mount which has nothing to do with
> Cassandra configuration:
> {noformat}
> df -h /
> Filesystem Size Used Avail Use% Mounted on
> 40G 5.7G 35G 15% /
> {noformat}
> I see this error running Cassandra 3.0.22
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
