Re: [DISCUSS] KIP-475: New Metric to Measure Number of Tasks on a Connector
Thanks, everyone, for the feedback. It looks like there's support, so I'll open the voting thread. Cyrus On Tue, Jun 4, 2019 at 9:02 PM Randall Hauch wrote: > Sorry that I was not clear. Yes, I was suggesting the state-specific counts > were in addition to the simple task count you originally proposed. Thanks > for taking my suggestion into account -- the updated KIP looks great. > > Thanks for contributing this improvement, Cyrus! > > Best regards, > > Randall > > On Tue, Jun 4, 2019 at 6:35 PM Cyrus Vafadari wrote: > > > Randall, > > > > I've updated the KIP to include all of your recommendations! > > > > Cyrus > > > > On Tue, Jun 4, 2019 at 2:55 PM Cyrus Vafadari > wrote: > > > > > Randall, > > > > > > I plan to update the public details section and the performance impact > as > > > you recommended. > > > > > > Regarding state-specific counts, I do agree this is a useful addition. > > > Before I make the change, I'd like to agree that these state-specific > > > counts should be in addition to the already-proposed total tasks count > > > (even though might be redundant, it is robust against new/missed > > connector > > > states, and is a useful metric in its own right), yes? > > > > > > Cyrus > > > > > > On Tue, Jun 4, 2019 at 12:24 PM Randall Hauch > wrote: > > > > > >> Thanks, Cyrus -- this will be quite useful. I do have a few > > >> comments/requests. > > >> > > >> Can you please be more specific about the public details about the > > metric? > > >> What is the MBean name on which the metric will appear? For example, > the > > >> AK > > >> documentation ( > > https://kafka.apache.org/documentation/#connect_monitoring > > >> ) > > >> defines all of the metrics an where they will appear, as does > > >> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework > > >> . > > >> > > >> Secondly, while a metric showing the total number of tasks is very > > useful, > > >> might it be worth considering also adding metrics for the number of > > >> running > > >> tasks, the number of paused tasks, and the number of failed tasks for > a > > >> connector. It might require using the herder's `connectorStatus(String > > >> connectorName)` method instead, but that appears to be just as > effective > > >> at > > >> using the local snapshot of the status store cache. > > >> > > >> Thirdly, it might be useful for the KIP to address the potential > > >> performance impact of computing these methods. Again, IIUC, the herder > > >> methods that the proposal mentions use the status and config stores > > caches > > >> only, so the impact should be negligible. > > >> > > >> Best regards, > > >> > > >> Randall > > >> > > >> On Sun, Jun 2, 2019 at 10:05 PM Ryanne Dolan > > >> wrote: > > >> > > >> > Cyrus, I agree this would be useful. > > >> > > > >> > Ryanne > > >> > > > >> > On Fri, May 31, 2019, 7:10 PM Oleksandr Diachenko < > > >> odiache...@apache.org> > > >> > wrote: > > >> > > > >> > > > > >> > > > > >> > > On 2019/05/30 06:06:12, Cyrus Vafadari > wrote: > > >> > > > Hello Dev, > > >> > > > > > >> > > > I'd like to start the discussion of KIP-475: New Metric to > Measure > > >> > Number > > >> > > > of Tasks on a Connector. > > >> > > > > > >> > > > > >> > > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-475%3A+New+Metric+to+Measure+Number+of+Tasks+on+a+Connector > > >> > > > > > >> > > > The proposal is pretty straightforward -- to add a new metric to > > >> > Connect > > >> > > to > > >> > > > measure the number of tasks on a Connector. Currently, we > support > > >> this > > >> > on > > >> > > > Worker level, so this KIP just adds another metric to support > this > > >> > > > per-connector. > > >> > > > > > >> > > > There is also a PR: > > >> > > > https://github.com/apache/kafka/pull/6843 > > >> > > > > > >> > > > Thanks, > > >> > > > > > >> > > > Cyrus > > >> > > > > > >> > > > > >> > > Hi Cyrus, > > >> > > > > >> > > That sounds like a useful addition. > > >> > > > > >> > > Regards, Alex. > > >> > > > > >> > > > >> > > > > > >
Re: [DISCUSS] KIP-475: New Metric to Measure Number of Tasks on a Connector
Sorry that I was not clear. Yes, I was suggesting the state-specific counts were in addition to the simple task count you originally proposed. Thanks for taking my suggestion into account -- the updated KIP looks great. Thanks for contributing this improvement, Cyrus! Best regards, Randall On Tue, Jun 4, 2019 at 6:35 PM Cyrus Vafadari wrote: > Randall, > > I've updated the KIP to include all of your recommendations! > > Cyrus > > On Tue, Jun 4, 2019 at 2:55 PM Cyrus Vafadari wrote: > > > Randall, > > > > I plan to update the public details section and the performance impact as > > you recommended. > > > > Regarding state-specific counts, I do agree this is a useful addition. > > Before I make the change, I'd like to agree that these state-specific > > counts should be in addition to the already-proposed total tasks count > > (even though might be redundant, it is robust against new/missed > connector > > states, and is a useful metric in its own right), yes? > > > > Cyrus > > > > On Tue, Jun 4, 2019 at 12:24 PM Randall Hauch wrote: > > > >> Thanks, Cyrus -- this will be quite useful. I do have a few > >> comments/requests. > >> > >> Can you please be more specific about the public details about the > metric? > >> What is the MBean name on which the metric will appear? For example, the > >> AK > >> documentation ( > https://kafka.apache.org/documentation/#connect_monitoring > >> ) > >> defines all of the metrics an where they will appear, as does > >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework > >> . > >> > >> Secondly, while a metric showing the total number of tasks is very > useful, > >> might it be worth considering also adding metrics for the number of > >> running > >> tasks, the number of paused tasks, and the number of failed tasks for a > >> connector. It might require using the herder's `connectorStatus(String > >> connectorName)` method instead, but that appears to be just as effective > >> at > >> using the local snapshot of the status store cache. > >> > >> Thirdly, it might be useful for the KIP to address the potential > >> performance impact of computing these methods. Again, IIUC, the herder > >> methods that the proposal mentions use the status and config stores > caches > >> only, so the impact should be negligible. > >> > >> Best regards, > >> > >> Randall > >> > >> On Sun, Jun 2, 2019 at 10:05 PM Ryanne Dolan > >> wrote: > >> > >> > Cyrus, I agree this would be useful. > >> > > >> > Ryanne > >> > > >> > On Fri, May 31, 2019, 7:10 PM Oleksandr Diachenko < > >> odiache...@apache.org> > >> > wrote: > >> > > >> > > > >> > > > >> > > On 2019/05/30 06:06:12, Cyrus Vafadari wrote: > >> > > > Hello Dev, > >> > > > > >> > > > I'd like to start the discussion of KIP-475: New Metric to Measure > >> > Number > >> > > > of Tasks on a Connector. > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-475%3A+New+Metric+to+Measure+Number+of+Tasks+on+a+Connector > >> > > > > >> > > > The proposal is pretty straightforward -- to add a new metric to > >> > Connect > >> > > to > >> > > > measure the number of tasks on a Connector. Currently, we support > >> this > >> > on > >> > > > Worker level, so this KIP just adds another metric to support this > >> > > > per-connector. > >> > > > > >> > > > There is also a PR: > >> > > > https://github.com/apache/kafka/pull/6843 > >> > > > > >> > > > Thanks, > >> > > > > >> > > > Cyrus > >> > > > > >> > > > >> > > Hi Cyrus, > >> > > > >> > > That sounds like a useful addition. > >> > > > >> > > Regards, Alex. > >> > > > >> > > >> > > >
Re: [DISCUSS] KIP-475: New Metric to Measure Number of Tasks on a Connector
Randall, I've updated the KIP to include all of your recommendations! Cyrus On Tue, Jun 4, 2019 at 2:55 PM Cyrus Vafadari wrote: > Randall, > > I plan to update the public details section and the performance impact as > you recommended. > > Regarding state-specific counts, I do agree this is a useful addition. > Before I make the change, I'd like to agree that these state-specific > counts should be in addition to the already-proposed total tasks count > (even though might be redundant, it is robust against new/missed connector > states, and is a useful metric in its own right), yes? > > Cyrus > > On Tue, Jun 4, 2019 at 12:24 PM Randall Hauch wrote: > >> Thanks, Cyrus -- this will be quite useful. I do have a few >> comments/requests. >> >> Can you please be more specific about the public details about the metric? >> What is the MBean name on which the metric will appear? For example, the >> AK >> documentation (https://kafka.apache.org/documentation/#connect_monitoring >> ) >> defines all of the metrics an where they will appear, as does >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework >> . >> >> Secondly, while a metric showing the total number of tasks is very useful, >> might it be worth considering also adding metrics for the number of >> running >> tasks, the number of paused tasks, and the number of failed tasks for a >> connector. It might require using the herder's `connectorStatus(String >> connectorName)` method instead, but that appears to be just as effective >> at >> using the local snapshot of the status store cache. >> >> Thirdly, it might be useful for the KIP to address the potential >> performance impact of computing these methods. Again, IIUC, the herder >> methods that the proposal mentions use the status and config stores caches >> only, so the impact should be negligible. >> >> Best regards, >> >> Randall >> >> On Sun, Jun 2, 2019 at 10:05 PM Ryanne Dolan >> wrote: >> >> > Cyrus, I agree this would be useful. >> > >> > Ryanne >> > >> > On Fri, May 31, 2019, 7:10 PM Oleksandr Diachenko < >> odiache...@apache.org> >> > wrote: >> > >> > > >> > > >> > > On 2019/05/30 06:06:12, Cyrus Vafadari wrote: >> > > > Hello Dev, >> > > > >> > > > I'd like to start the discussion of KIP-475: New Metric to Measure >> > Number >> > > > of Tasks on a Connector. >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-475%3A+New+Metric+to+Measure+Number+of+Tasks+on+a+Connector >> > > > >> > > > The proposal is pretty straightforward -- to add a new metric to >> > Connect >> > > to >> > > > measure the number of tasks on a Connector. Currently, we support >> this >> > on >> > > > Worker level, so this KIP just adds another metric to support this >> > > > per-connector. >> > > > >> > > > There is also a PR: >> > > > https://github.com/apache/kafka/pull/6843 >> > > > >> > > > Thanks, >> > > > >> > > > Cyrus >> > > > >> > > >> > > Hi Cyrus, >> > > >> > > That sounds like a useful addition. >> > > >> > > Regards, Alex. >> > > >> > >> >
Re: [DISCUSS] KIP-475: New Metric to Measure Number of Tasks on a Connector
Randall, I plan to update the public details section and the performance impact as you recommended. Regarding state-specific counts, I do agree this is a useful addition. Before I make the change, I'd like to agree that these state-specific counts should be in addition to the already-proposed total tasks count (even though might be redundant, it is robust against new/missed connector states, and is a useful metric in its own right), yes? Cyrus On Tue, Jun 4, 2019 at 12:24 PM Randall Hauch wrote: > Thanks, Cyrus -- this will be quite useful. I do have a few > comments/requests. > > Can you please be more specific about the public details about the metric? > What is the MBean name on which the metric will appear? For example, the AK > documentation (https://kafka.apache.org/documentation/#connect_monitoring) > defines all of the metrics an where they will appear, as does > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework > . > > Secondly, while a metric showing the total number of tasks is very useful, > might it be worth considering also adding metrics for the number of running > tasks, the number of paused tasks, and the number of failed tasks for a > connector. It might require using the herder's `connectorStatus(String > connectorName)` method instead, but that appears to be just as effective at > using the local snapshot of the status store cache. > > Thirdly, it might be useful for the KIP to address the potential > performance impact of computing these methods. Again, IIUC, the herder > methods that the proposal mentions use the status and config stores caches > only, so the impact should be negligible. > > Best regards, > > Randall > > On Sun, Jun 2, 2019 at 10:05 PM Ryanne Dolan > wrote: > > > Cyrus, I agree this would be useful. > > > > Ryanne > > > > On Fri, May 31, 2019, 7:10 PM Oleksandr Diachenko > > > wrote: > > > > > > > > > > > On 2019/05/30 06:06:12, Cyrus Vafadari wrote: > > > > Hello Dev, > > > > > > > > I'd like to start the discussion of KIP-475: New Metric to Measure > > Number > > > > of Tasks on a Connector. > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-475%3A+New+Metric+to+Measure+Number+of+Tasks+on+a+Connector > > > > > > > > The proposal is pretty straightforward -- to add a new metric to > > Connect > > > to > > > > measure the number of tasks on a Connector. Currently, we support > this > > on > > > > Worker level, so this KIP just adds another metric to support this > > > > per-connector. > > > > > > > > There is also a PR: > > > > https://github.com/apache/kafka/pull/6843 > > > > > > > > Thanks, > > > > > > > > Cyrus > > > > > > > > > > Hi Cyrus, > > > > > > That sounds like a useful addition. > > > > > > Regards, Alex. > > > > > >
Re: [DISCUSS] KIP-475: New Metric to Measure Number of Tasks on a Connector
Thanks, Cyrus -- this will be quite useful. I do have a few comments/requests. Can you please be more specific about the public details about the metric? What is the MBean name on which the metric will appear? For example, the AK documentation (https://kafka.apache.org/documentation/#connect_monitoring) defines all of the metrics an where they will appear, as does https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework . Secondly, while a metric showing the total number of tasks is very useful, might it be worth considering also adding metrics for the number of running tasks, the number of paused tasks, and the number of failed tasks for a connector. It might require using the herder's `connectorStatus(String connectorName)` method instead, but that appears to be just as effective at using the local snapshot of the status store cache. Thirdly, it might be useful for the KIP to address the potential performance impact of computing these methods. Again, IIUC, the herder methods that the proposal mentions use the status and config stores caches only, so the impact should be negligible. Best regards, Randall On Sun, Jun 2, 2019 at 10:05 PM Ryanne Dolan wrote: > Cyrus, I agree this would be useful. > > Ryanne > > On Fri, May 31, 2019, 7:10 PM Oleksandr Diachenko > wrote: > > > > > > > On 2019/05/30 06:06:12, Cyrus Vafadari wrote: > > > Hello Dev, > > > > > > I'd like to start the discussion of KIP-475: New Metric to Measure > Number > > > of Tasks on a Connector. > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-475%3A+New+Metric+to+Measure+Number+of+Tasks+on+a+Connector > > > > > > The proposal is pretty straightforward -- to add a new metric to > Connect > > to > > > measure the number of tasks on a Connector. Currently, we support this > on > > > Worker level, so this KIP just adds another metric to support this > > > per-connector. > > > > > > There is also a PR: > > > https://github.com/apache/kafka/pull/6843 > > > > > > Thanks, > > > > > > Cyrus > > > > > > > Hi Cyrus, > > > > That sounds like a useful addition. > > > > Regards, Alex. > > >
Re: [DISCUSS] KIP-475: New Metric to Measure Number of Tasks on a Connector
Cyrus, I agree this would be useful. Ryanne On Fri, May 31, 2019, 7:10 PM Oleksandr Diachenko wrote: > > > On 2019/05/30 06:06:12, Cyrus Vafadari wrote: > > Hello Dev, > > > > I'd like to start the discussion of KIP-475: New Metric to Measure Number > > of Tasks on a Connector. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-475%3A+New+Metric+to+Measure+Number+of+Tasks+on+a+Connector > > > > The proposal is pretty straightforward -- to add a new metric to Connect > to > > measure the number of tasks on a Connector. Currently, we support this on > > Worker level, so this KIP just adds another metric to support this > > per-connector. > > > > There is also a PR: > > https://github.com/apache/kafka/pull/6843 > > > > Thanks, > > > > Cyrus > > > > Hi Cyrus, > > That sounds like a useful addition. > > Regards, Alex. >
Re: [DISCUSS] KIP-475: New Metric to Measure Number of Tasks on a Connector
On 2019/05/30 06:06:12, Cyrus Vafadari wrote: > Hello Dev, > > I'd like to start the discussion of KIP-475: New Metric to Measure Number > of Tasks on a Connector. > https://cwiki.apache.org/confluence/display/KAFKA/KIP-475%3A+New+Metric+to+Measure+Number+of+Tasks+on+a+Connector > > The proposal is pretty straightforward -- to add a new metric to Connect to > measure the number of tasks on a Connector. Currently, we support this on > Worker level, so this KIP just adds another metric to support this > per-connector. > > There is also a PR: > https://github.com/apache/kafka/pull/6843 > > Thanks, > > Cyrus > Hi Cyrus, That sounds like a useful addition. Regards, Alex.
[DISCUSS] KIP-475: New Metric to Measure Number of Tasks on a Connector
Hello Dev, I'd like to start the discussion of KIP-475: New Metric to Measure Number of Tasks on a Connector. https://cwiki.apache.org/confluence/display/KAFKA/KIP-475%3A+New+Metric+to+Measure+Number+of+Tasks+on+a+Connector The proposal is pretty straightforward -- to add a new metric to Connect to measure the number of tasks on a Connector. Currently, we support this on Worker level, so this KIP just adds another metric to support this per-connector. There is also a PR: https://github.com/apache/kafka/pull/6843 Thanks, Cyrus