[jira] [Commented] (CASSANDRA-10323) Add more MaterializedView metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14939806#comment-14939806 ] Chris Lohfink commented on CASSANDRA-10323: --- Changes look good to me > Add more MaterializedView metrics > - > > Key: CASSANDRA-10323 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10323 > Project: Cassandra > Issue Type: Improvement >Reporter: T Jake Luciani >Assignee: Chris Lohfink > Fix For: 3.0.0 rc2 > > Attachments: trunk-10323-v2.txt, trunk-10323.txt > > > We need to add more metrics to help understand where time is spent in > materialized view writes. We currently track the ratio of async base -> view > mutations that fail. > We should also add > * The amount of time spent waiting for the partition lock (contention) > * The amount of time spent reading data > Any others? > [~carlyeks] [~jkni] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10323) Add more MaterializedView metrics
[
https://issues.apache.org/jira/browse/CASSANDRA-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14938771#comment-14938771
]
Carl Yeksigian commented on CASSANDRA-10323:
Overall +1. I pushed [a
branch|https://github.com/carlyeks/cassandra/tree/review/10323/3.0] with just a
few nits; if you could take a look at them [~cnlwsu].
* Removed the overload of {{mutateMV}} since we haven't hit a stable release
with MV in yet.
* Don't initialize the {{viewLockAcquire}} and {{viewRead}} metrics if this is
a view. I was expecting those values to be updated when I was looking at the
metrics; probably makes sense to just not have them at all for views instead of
having the unused metrics.
* Formatting miscellany
> Add more MaterializedView metrics
> -
>
> Key: CASSANDRA-10323
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10323
> Project: Cassandra
> Issue Type: Improvement
>Reporter: T Jake Luciani
>Assignee: Chris Lohfink
> Fix For: 3.0.0 rc2
>
> Attachments: trunk-10323-v2.txt, trunk-10323.txt
>
>
> We need to add more metrics to help understand where time is spent in
> materialized view writes. We currently track the ratio of async base -> view
> mutations that fail.
> We should also add
> * The amount of time spent waiting for the partition lock (contention)
> * The amount of time spent reading data
> Any others?
> [~carlyeks] [~jkni]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10323) Add more MaterializedView metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14934121#comment-14934121 ] Chris Lohfink commented on CASSANDRA-10323: --- have it recording time from when base mutation is applied to memtable until when the CL.ONE is achieved on the async write to the MVs (I think). Added a "pending", but its global not per table. Also tracking per table how much time is spent on the local read and how much time it takes to acquire partition lock. > Add more MaterializedView metrics > - > > Key: CASSANDRA-10323 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10323 > Project: Cassandra > Issue Type: Improvement >Reporter: T Jake Luciani >Assignee: Chris Lohfink > Labels: lhf > Fix For: 3.0.0 rc2 > > Attachments: trunk-10323-v2.txt, trunk-10323.txt > > > We need to add more metrics to help understand where time is spent in > materialized view writes. We currently track the ratio of async base -> view > mutations that fail. > We should also add > * The amount of time spent waiting for the partition lock (contention) > * The amount of time spent reading data > Any others? > [~carlyeks] [~jkni] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10323) Add more MaterializedView metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14908951#comment-14908951 ] Chris Lohfink commented on CASSANDRA-10323: --- It's measure of how many mutations are outstanding to replicas of the MVs partitions after/during the base replicas mutation > Add more MaterializedView metrics > - > > Key: CASSANDRA-10323 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10323 > Project: Cassandra > Issue Type: Improvement >Reporter: T Jake Luciani > Labels: lhf > Fix For: 3.0.0 rc2 > > Attachments: trunk-10323.txt > > > We need to add more metrics to help understand where time is spent in > materialized view writes. We currently track the ratio of async base -> view > mutations that fail. > We should also add > * The amount of time spent waiting for the partition lock (contention) > * The amount of time spent reading data > Any others? > [~carlyeks] [~jkni] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10323) Add more MaterializedView metrics
[
https://issues.apache.org/jira/browse/CASSANDRA-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14908782#comment-14908782
]
Robert Coli commented on CASSANDRA-10323:
-
{quote}Is taking the attempted replicas - successful replicas work for that?
Can make a gauge to make it easier{quote}
That's a measure of how many retries you've ever had, not a measure of how many
retries are in the queue now, isn't it? To me the latter is more valuable than
the former.
> Add more MaterializedView metrics
> -
>
> Key: CASSANDRA-10323
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10323
> Project: Cassandra
> Issue Type: Improvement
>Reporter: T Jake Luciani
> Labels: lhf
> Fix For: 3.0.0 rc2
>
> Attachments: trunk-10323.txt
>
>
> We need to add more metrics to help understand where time is spent in
> materialized view writes. We currently track the ratio of async base -> view
> mutations that fail.
> We should also add
> * The amount of time spent waiting for the partition lock (contention)
> * The amount of time spent reading data
> Any others?
> [~carlyeks] [~jkni]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10323) Add more MaterializedView metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14908742#comment-14908742 ] Chris Lohfink commented on CASSANDRA-10323: --- Is taking the attempted replicas - successful replicas work for that? Can make a gauge to make it easier > Add more MaterializedView metrics > - > > Key: CASSANDRA-10323 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10323 > Project: Cassandra > Issue Type: Improvement >Reporter: T Jake Luciani > Labels: lhf > Fix For: 3.0.0 rc2 > > Attachments: trunk-10323.txt > > > We need to add more metrics to help understand where time is spent in > materialized view writes. We currently track the ratio of async base -> view > mutations that fail. > We should also add > * The amount of time spent waiting for the partition lock (contention) > * The amount of time spent reading data > Any others? > [~carlyeks] [~jkni] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10323) Add more MaterializedView metrics
[
https://issues.apache.org/jira/browse/CASSANDRA-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14908670#comment-14908670
]
Robert Coli commented on CASSANDRA-10323:
-
{quote}Robert Coli raised the good point that a very helpful metric for
operators would be to track MV lag, because mutations are applied to the view
from the base table asynchronously.{quote}
Probably this ends up being expressed as "queue depth" as opposed to actual
temporal lag, but "queue depth" provides useful visibility into "lag."
> Add more MaterializedView metrics
> -
>
> Key: CASSANDRA-10323
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10323
> Project: Cassandra
> Issue Type: Improvement
>Reporter: T Jake Luciani
> Labels: lhf
> Fix For: 3.0.0 rc2
>
> Attachments: trunk-10323.txt
>
>
> We need to add more metrics to help understand where time is spent in
> materialized view writes. We currently track the ratio of async base -> view
> mutations that fail.
> We should also add
> * The amount of time spent waiting for the partition lock (contention)
> * The amount of time spent reading data
> Any others?
> [~carlyeks] [~jkni]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10323) Add more MaterializedView metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906950#comment-14906950 ] Chris Lohfink commented on CASSANDRA-10323: --- Ill give it a try > Add more MaterializedView metrics > - > > Key: CASSANDRA-10323 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10323 > Project: Cassandra > Issue Type: Improvement >Reporter: T Jake Luciani > Labels: lhf > Fix For: 3.0.0 rc2 > > > We need to add more metrics to help understand where time is spent in > materialized view writes. We currently track the ratio of async base -> view > mutations that fail. > We should also add > * The amount of time spent waiting for the partition lock (contention) > * The amount of time spent reading data > Any others? > [~carlyeks] [~jkni] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10323) Add more MaterializedView metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906749#comment-14906749 ] Philip Thompson commented on CASSANDRA-10323: - [~rcoli] raised the good point that a very helpful metric for operators would be to track MV lag, because mutations are applied to the view from the base table asynchronously. > Add more MaterializedView metrics > - > > Key: CASSANDRA-10323 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10323 > Project: Cassandra > Issue Type: Improvement >Reporter: T Jake Luciani > Labels: lhf > Fix For: 3.0.0 rc2 > > > We need to add more metrics to help understand where time is spent in > materialized view writes. We currently track the ratio of async base -> view > mutations that fail. > We should also add > * The amount of time spent waiting for the partition lock (contention) > * The amount of time spent reading data > Any others? > [~carlyeks] [~jkni] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10323) Add more MaterializedView metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14903376#comment-14903376 ] Brandon Williams commented on CASSANDRA-10323: -- Interested, [~cnlwsu]? > Add more MaterializedView metrics > - > > Key: CASSANDRA-10323 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10323 > Project: Cassandra > Issue Type: Improvement >Reporter: T Jake Luciani > Labels: lhf > Fix For: 3.0.0 rc2 > > > We need to add more metrics to help understand where time is spent in > materialized view writes. We currently track the ratio of async base -> view > mutations that fail. > We should also add > * The amount of time spent waiting for the partition lock (contention) > * The amount of time spent reading data > Any others? > [~carlyeks] [~jkni] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10323) Add more MaterializedView metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745557#comment-14745557 ] Joel Knighton commented on CASSANDRA-10323: --- Agreed re: paragraph 3. I'll watch that closer in the future. > Add more MaterializedView metrics > - > > Key: CASSANDRA-10323 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10323 > Project: Cassandra > Issue Type: Improvement >Reporter: T Jake Luciani >Assignee: T Jake Luciani > Fix For: 3.0.0 rc1 > > > We need to add more metrics to help understand where time is spent in > materialized view writes. We currently track the ratio of async base -> view > mutations that fail. > We should also add > * The amount of time spent waiting for the partition lock (contention) > * The amount of time spent reading data > Any others? > [~carlyeks] [~jkni] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10323) Add more MaterializedView metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745554#comment-14745554 ] Aleksey Yeschenko commented on CASSANDRA-10323: --- It's possible to do that quite efficiently now that CASSANDRA-9673 is in. Commented about it [here|https://issues.apache.org/jira/browse/CASSANDRA-9673?focusedCommentId=14712205&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14712205]. That's metrics. Batchlog however is not a mechanism for achieving quorum writes and shouldn't be seen as such. > Add more MaterializedView metrics > - > > Key: CASSANDRA-10323 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10323 > Project: Cassandra > Issue Type: Improvement >Reporter: T Jake Luciani >Assignee: T Jake Luciani > Fix For: 3.0.0 rc1 > > > We need to add more metrics to help understand where time is spent in > materialized view writes. We currently track the ratio of async base -> view > mutations that fail. > We should also add > * The amount of time spent waiting for the partition lock (contention) > * The amount of time spent reading data > Any others? > [~carlyeks] [~jkni] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10323) Add more MaterializedView metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745548#comment-14745548 ] Joel Knighton commented on CASSANDRA-10323: --- I'd be interested to see counters for batchlog entries created/removed by a coordinator. Effectively, how often does batchlog replay kick in as a mechanism to work toward quorum writes? > Add more MaterializedView metrics > - > > Key: CASSANDRA-10323 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10323 > Project: Cassandra > Issue Type: Improvement >Reporter: T Jake Luciani >Assignee: T Jake Luciani > Fix For: 3.0.0 rc1 > > > We need to add more metrics to help understand where time is spent in > materialized view writes. We currently track the ratio of async base -> view > mutations that fail. > We should also add > * The amount of time spent waiting for the partition lock (contention) > * The amount of time spent reading data > Any others? > [~carlyeks] [~jkni] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
