I am also happy with max/min/99/90. And I buy your naming argument about
staleness vs latency.


-Matthias

On 5/15/20 12:24 PM, Boyang Chen wrote:
> Hey Sophie,
> 
> 90/99/min/max make sense to me.
> 
> On Fri, May 15, 2020 at 12:20 PM Sophie Blee-Goldman <sop...@confluent.io>
> wrote:
> 
>> @Matthias
>> Regarding tracking the 50th percentile, I'll refer you to the 4:53 mark of
>> the video
>> *you* linked: https://youtu.be/lJ8ydIuPFeU?t=293
>>
>> And that's what he says about the 95th percentile! Imagine what he would
>> say about
>> the 50th :P
>>
>> But seriously, since we can't seem to agree that the mean or 50th
>> percentile is actually
>> useful I'm inclined to resurrect my original proposal, neither. But I think
>> that's a good
>> argument against the 75th, which I admittedly chose somewhat arbitrarily as
>> an
>> intermediate between the 50th and the higher percentiles. How about:
>>
>> -max
>> -p99
>> -p90
>> -min
>>
>> with p50/mean still up for debate if anyone feels strongly for either of
>> them.
>>
>> Regarding the name, I've already flip-flopped on this so I'm definitely
>> still open to
>> further arguments. But the reason for changing it from end-to-end-latency
>> (which
>> is similar to what you propose) is that this metric technically reflects
>> how old (ie how "stale")
>> the record is when it's *received* by the operator, not when it's processed
>> by the operator.
>> It seemed like there was the potential for confusion that
>> "end-to-end-latency" might
>> represent the latency from the event creation to the time the processor is
>> done
>> processing it.
>>
>> @John
>> I'd rather err on the side of "not-enough" metrics as we can always add
>> this to the
>> stateless metrics later on. If we decide to measure the time at every node
>> and don't
>> find any evidence of a serious performance impact, and users indicate they
>> would
>> like to see this metric at all nodes, then we can easily start reporting
>> them as well.
>> WDYT?
>>
>> That said, sink nodes seem like a reasonable exception to the rule.
>> Obviously users
>> should be able to detect the time when the record reaches the output topic
>> but that
>> still leaves a gap in understanding how long the production latency was.
>> This mirrors
>> the consumption latency that is exposed by the task-level metrics, which
>> are measured
>> at the source node. For good symmetry what if we actually expose both the
>> source
>> and sink latency at the task-level? ie report both sets of statistical
>> measurements with
>> the additional tag -source/-sink
>>
>> @Bill
>> Thanks for the comment about regarding the min! I hadn't considered that
>> and it's
>> quite useful to think about how and what is useful from a users point of
>> view.
>>
>> Regarding your second. point, I'm inclined to leave that as an
>> implementation detail
>> but my take would be that the user should be allowed to control the record
>> timestamp
>> used for this with the timestamp extractor. My impression is that users may
>> often embed
>> the actual event time in the payload for whatever reason, and this
>> represents the "true"
>> timestamp as far as the Streams topology is concerned.
>>
>>
>> On Fri, May 15, 2020 at 11:05 AM Bill Bejeck <bbej...@gmail.com> wrote:
>>
>>> Thanks for the KIP, Sophie, this will be a useful metric to add.
>>>
>>> Regarding tracking min, I  think it could be valuable for users to
>> discern
>>> which part of their topologies are more efficient since this is a
>>> task-level metric.  I realize everyone seems to be on board with
>> including
>>> min anyway, but I wanted to add my 2 cents on this topic should we decide
>>> to revisit adding min or not.
>>>
>>> I do have a question regarding the calculation of staleness.
>>> Is there going to be a consideration for timestamp extractors? Users
>> could
>>> prefer to use a timestamp embedded in the payload, and it could skew the
>>> measurements.
>>> I was wondering if we should specify in the KIP if setting the arrival
>> time
>>> is always going to come from the record timestamp, or is this an
>>> implementation detail we can cover in the PR?
>>>
>>> Thanks!
>>> Bill
>>>
>>> On Fri, May 15, 2020 at 1:11 AM Matthias J. Sax <mj...@apache.org>
>> wrote:
>>>
>>>> Thanks for the KIP Sophie.
>>>>
>>>> I think it's not useful to record the avg/mean; it sensitive to
>>>> outliers. We should rather track the median (50th percentile).
>>>>
>>>> Not sure if tracking min is useful, but I am also ok to track it.
>>>>
>>>> However, I find it odd to track 75th percentile. Standard measures
>> would
>>>> the 90th or 95th -- I guess we don't need both, so maybe picking 90th
>>>> might be more useful?
>>>>
>>>> About the name: "staleness" wound really odd, and if fact the metric
>>>> does capture "latency" so we should call it "latency". I understand the
>>>> issue that we already have a latency metric. So maybe we could call it
>>>> `record-e2e-latency-*` ?
>>>>
>>>> While I agree that we should include out-or-order data (the KIP should
>>>> talk about `out-of-order` data, not `late` data; data is only `late` if
>>>> it's out-of-order and if it's dropped), I don't really understand why
>>>> the new metric would help to configure grace period or retention time?
>>>> As you mention in the KIP, both are define as max difference of
>>>> `event-time - stream-time` and thus the new metric that takes
>>>> system-/wallclock-time into account does not seem to help at all.
>>>>
>>>>
>>>> Btw: there is a great talk about "How NOT to Measure Latency" by Gil
>>>> Tene: https://www.youtube.com/watch?v=lJ8ydIuPFeU
>>>>
>>>>
>>>> -Matthias
>>>>
>>>>
>>>> On 5/14/20 7:17 PM, John Roesler wrote:
>>>>> Hi Sophie,
>>>>>
>>>>> It seems like there would still be plenty of use cases for recording
>>>>> this metric at all processors and not just stateful ones, but I'm
>> happy
>>>>> to suspend my arguments for now. Since you're proposing to keep
>>>>> them at the processor-node level, it will be seamless later to add
>>>>> in the stateless processors if we want. As a wise man once said,
>>>>> "Adding is always easier than removing."
>>>>>
>>>>> Regarding the time measurement, it's an implementation detail
>>>>> we don't need to consider in the KIP. Nevertheless, I'd greatly
>>>>> prefer to measure the system time again when recording the
>>>>> metric. I don't think we've seen any evidence that proves this
>>>>> would harm performance, and the amount of inaccuracy using
>>>>> the cached system time could incur is actually substantial. But,
>>>>> if you want to just "not mention this" in the KIP, we can defer to
>>>>> the actual PR discussion, at which time we're in a better position
>>>>> to use benchmarks, etc., to make the call.
>>>>>
>>>>> Along the lines of the measurement accuracy discussion, one
>>>>> minor thought I had is that maybe we should consider measuring
>>>>> the task staleness metric at the sink, rather than the source, so
>> that
>>>>> it includes the processing latency of the task itself, not just the
>>>> latency
>>>>> of everything up to, but not including, the task (which seems
>> confusing
>>>>> for users). I guess this could also be an implementation detail,
>>> though.
>>>>>
>>>>> Thanks for the update,
>>>>> -John
>>>>>
>>>>> On Thu, May 14, 2020, at 13:31, Sophie Blee-Goldman wrote:
>>>>>> Hey all,
>>>>>>
>>>>>> After discussing with Bruno I'd like to propose a small amendment,
>>>>>> which is to record the processor-node-level metrics only for
>>> *stateful*
>>>>>> *operators*. They would still be considered a "processor-node-level"
>>>>>> metric and not a "state-store-level" metric as the staleness is
>> still
>>>>>> a property of the node rather than of the state itself. However, it
>>>> seems
>>>>>> that this information is primarily useful for stateful operators
>> that
>>>> might
>>>>>> be exposing state via IQ or otherwise dependent on the record time
>>>>>> unlike a stateless operator.
>>>>>>
>>>>>> It's worth calling out that recent performance improvements to the
>>>> metrics
>>>>>> framework mean that we no longer fetch the system time at the
>> operator
>>>>>> level, but only once per task. In other words the system time is not
>>>> updated
>>>>>> between each process as a record flows through the subtopology, so
>>>>>> debugging the processor-level latency via the stateleness will not
>> be
>>>>>> possible.Note that this doesn't mean the operator-level metrics are
>>> not
>>>>>> *useful* relative to the task-level metric. Upstream caching and/or
>>>>>> suppression
>>>>>> can still cause a record's staleness at some downstream stateful
>>>> operator
>>>>>> to deviate from the task-level staleness (recorded at the source
>>> node).
>>>>>>
>>>>>> Please let me know if you have any concerns about this change. The
>>>>>> KIP has been updated with the new proposal
>>>>>>
>>>>>> On Thu, May 14, 2020 at 3:04 AM Bruno Cadonna <br...@confluent.io>
>>>> wrote:
>>>>>>
>>>>>>> Hi Sophie,
>>>>>>>
>>>>>>> Thank you for the KIP.
>>>>>>>
>>>>>>> The KIP looks good to me.
>>>>>>>
>>>>>>> 50th percentile:
>>>>>>> I think we do not need it now. If we need it, we can add it. Here
>> the
>>>>>>> old truism applies: Adding is always easier than removing.
>>>>>>>
>>>>>>> processor-node-level metrics:
>>>>>>> I think it is good to have the staleness metrics also on
>>>>>>> processor-node-level. If we do not want to record them on all
>>>>>>> processor nodes, you could restrict the recording to stateful
>>>>>>> processor-nodes, since those are the ones that would benefit most
>>> from
>>>>>>> the staleness metrics.
>>>>>>>
>>>>>>> Best,
>>>>>>> Bruno
>>>>>>>
>>>>>>> On Thu, May 14, 2020 at 4:15 AM Sophie Blee-Goldman <
>>>> sop...@confluent.io>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Yeah, the specific reason was just to align with the current
>>> metrics.
>>>>>>>>
>>>>>>>> Is it better to conform than to be right? History has a lot to say
>>> on
>>>>>>> that
>>>>>>>> matter
>>>>>>>> but I'm not sure how much of it applies to the fine details of
>>> metrics
>>>>>>>> naming :P
>>>>>>>>
>>>>>>>> More seriously, I figured if people are looking at this metric
>>> they're
>>>>>>>> likely to
>>>>>>>> be looking at all the others. Then naming this one "-mean" would
>>>> probably
>>>>>>>> lead some to conclude that the "-avg" suffix in the other metrics
>>> has
>>>> a
>>>>>>>> different meaning.
>>>>>>>>
>>>>>>>> As for the percentiles, I actually like p99 (and p75) better. I'll
>>>> swap
>>>>>>>> that out
>>>>>>>>
>>>>>>>> On Wed, May 13, 2020 at 7:07 PM John Roesler <vvcep...@apache.org
>>>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks Sophie,
>>>>>>>>>
>>>>>>>>> I hope this isn't too nit-picky, but is there a reason to choose
>>>> "avg"
>>>>>>>>> instead
>>>>>>>>> of "mean"? Maybe this is too paranoid, and I might be
>> oversensitive
>>>>>>> because
>>>>>>>>> of the mistake I just made earlier, but it strikes me that "avg"
>> is
>>>>>>>>> actually
>>>>>>>>> ambiguous, as it refers to a family of statistics, whereas "mean"
>>> is
>>>>>>>>> specific.
>>>>>>>>> I see other Kafka metrics with "avg", but none with "mean"; was
>>> that
>>>>>>> the
>>>>>>>>> reason? If so, I'm +1.
>>>>>>>>>
>>>>>>>>> Regarding the names of the percentile, I actually couldn't find
>>> _any_
>>>>>>> other
>>>>>>>>> metrics that use percentile. Was there a reason to choose "99th"
>> as
>>>>>>> opposed
>>>>>>>>> to "p99" or any other scheme? This is not a criticism, I'm just
>>>>>>> primarily
>>>>>>>>> asking
>>>>>>>>> for consistency's sake.
>>>>>>>>>
>>>>>>>>> Thanks again,
>>>>>>>>> -John
>>>>>>>>>
>>>>>>>>> On Wed, May 13, 2020, at 19:19, Sophie Blee-Goldman wrote:
>>>>>>>>>> Alright, I can get behind adding the min metric for the sake of
>>>>>>> pretty
>>>>>>>>>> graphs
>>>>>>>>>> (and trivial computation).
>>>>>>>>>>
>>>>>>>>>> I'm still on the fence regarding the mean (or 50th percentile)
>>> but I
>>>>>>> can
>>>>>>>>> see
>>>>>>>>>> how users might expect it and find it a bit disorienting not to
>>>>>>> have. So
>>>>>>>>> the
>>>>>>>>>> updated proposed metrics are
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>    - record-staleness-max [ms]
>>>>>>>>>>    - record-staleness-99th [ms] *(99th percentile)*
>>>>>>>>>>    - record-staleness-75th [ms] *(75th percentile)*
>>>>>>>>>>    - record-staleness-avg [ms] *(mean)*
>>>>>>>>>>    - record-staleness-min [ms]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, May 13, 2020 at 4:42 PM John Roesler <
>> vvcep...@apache.org
>>>>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Oh boy, I never miss an opportunity to embarrass myself. I
>> guess
>>>>>>> the
>>>>>>>>> mean
>>>>>>>>>>> seems more interesting to me than the median, but neither are
>> as
>>>>>>>>>>> interesting as the higher percentiles (99th and max).
>>>>>>>>>>>
>>>>>>>>>>> Min isn’t really important for any SLAs, but it does round out
>>> the
>>>>>>>>> mental
>>>>>>>>>>> picture of the distribution. I’ve always graphed min along with
>>> the
>>>>>>>>> other
>>>>>>>>>>> metrics to help me understand how fast the system can be, which
>>>>>>> helps
>>>>>>>>> in
>>>>>>>>>>> optimization decisions. It’s also a relatively inexpensive
>> metric
>>>>>>> to
>>>>>>>>>>> compute, so it might be nice to just throw it in.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 13, 2020, at 18:18, Sophie Blee-Goldman wrote:
>>>>>>>>>>>> G1:
>>>>>>>>>>>> I was considering it as the "end-to-end latency *up* to the
>>>>>>> specific
>>>>>>>>>>> task"
>>>>>>>>>>>> but
>>>>>>>>>>>> I'm happy with "record-staleness" if that drives the point
>> home
>>>>>>>>> better.
>>>>>>>>>>> So
>>>>>>>>>>>> it's the
>>>>>>>>>>>> "staleness of the record when it is received by that task" --
>>>>>>> will
>>>>>>>>> update
>>>>>>>>>>>> the KIP
>>>>>>>>>>>>
>>>>>>>>>>>> B1/J:
>>>>>>>>>>>> I'm struggling to imagine a case where the min would actually
>> be
>>>>>>>>> useful,
>>>>>>>>>>>> rather than
>>>>>>>>>>>> just intellectually interesting. I don't feel strongly that we
>>>>>>>>> shouldn't
>>>>>>>>>>>> add it, but that's
>>>>>>>>>>>> why I didn't include it from the start. Can you enlighten me
>>>>>>> with an
>>>>>>>>>>>> example?
>>>>>>>>>>>>
>>>>>>>>>>>> I was also vaguely concerned about the overhead of adding
>>>>>>> multiple
>>>>>>>>>>>> percentile
>>>>>>>>>>>> metrics. Do we have any data to indicate what kind of
>>> performance
>>>>>>>>> hit we
>>>>>>>>>>>> take on
>>>>>>>>>>>> metrics computation?
>>>>>>>>>>>>
>>>>>>>>>>>> Also, not to be too pedantic but the 50th percentile would be
>>> the
>>>>>>>>> median
>>>>>>>>>>>> not the
>>>>>>>>>>>> mean. Would you propose to add the mean *and* the 50th
>>>>>>> percentile, or
>>>>>>>>>>> just
>>>>>>>>>>>> one
>>>>>>>>>>>> of the two?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks all!
>>>>>>>>>>>> Sophie
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, May 13, 2020 at 3:34 PM John Roesler <
>>>>>>> vvcep...@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello all, and thanks for the KIP, Sophie,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Just some comments on the discussion so far:
>>>>>>>>>>>>>
>>>>>>>>>>>>> B2/G1:
>>>>>>>>>>>>> In principle, it shouldn't matter whether we report "spans"
>> or
>>>>>>>>>>>>> "end-to-end" latency. But in practice, some of the spans are
>>>>>>> pretty
>>>>>>>>>>>>> difficult to really measure (like time spent waiting in the
>>>>>>>>> topics, or
>>>>>>>>>>>>> time from the event happening to the ETL producer choosing to
>>>>>>> send
>>>>>>>>> it,
>>>>>>>>>>>>> or time spent in send/receive buffers, etc., etc.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In other words, it's practically easier to compute spans by
>>>>>>>>> subtracting
>>>>>>>>>>>>> e2e latencies than it is to compute e2e latencies by adding
>>>>>>> spans.
>>>>>>>>> You
>>>>>>>>>>>>> can even just consider that the span computation from e2e
>>>>>>> always
>>>>>>>>> just
>>>>>>>>>>>>> involves subtracting two numbers, whereas computing e2e
>> latency
>>>>>>>>> from
>>>>>>>>>>>>> spans involves adding _all_ the spans leading up to the end
>> you
>>>>>>>>> care
>>>>>>>>>>> about.
>>>>>>>>>>>>>
>>>>>>>>>>>>> It seems like people really prefer to have spans when they
>> are
>>>>>>>>>>> debugging
>>>>>>>>>>>>> latency problems, whereas e2e latency is a more general
>>>>>>> measurement
>>>>>>>>>>>>> that basically every person/application cares about and
>> should
>>>>>>> be
>>>>>>>>>>>>> monitoring.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Altogether, it really seem to provide more value to more
>>>>>>> people if
>>>>>>>>> we
>>>>>>>>>>>>> report
>>>>>>>>>>>>> e2e latencies. Regarding "record-staleness" as a name, I
>> think
>>>>>>> I
>>>>>>>>> have
>>>>>>>>>>> no
>>>>>>>>>>>>> preference, I'd defer to other peoples' intuition.
>>>>>>>>>>>>>
>>>>>>>>>>>>> G2:
>>>>>>>>>>>>> I think the processor-node metric is nice, since the inside
>> of
>>>>>>> a
>>>>>>>>> task
>>>>>>>>>>> can
>>>>>>>>>>>>> introduce a significant amount of latency in some cases.
>> Plus,
>>>>>>>>> it's a
>>>>>>>>>>> more
>>>>>>>>>>>>> direct measurement, if you really wanted to know (for the
>>>>>>> purposes
>>>>>>>>> of
>>>>>>>>>>> IQ
>>>>>>>>>>>>> or something) how long it takes source events to "show up" at
>>>>>>> the
>>>>>>>>>>> store.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think actually recording it at every processor could be
>>>>>>>>> expensive,
>>>>>>>>>>> but we
>>>>>>>>>>>>> already record a bunch of metrics at the node level.
>>>>>>>>>>>>>
>>>>>>>>>>>>> B1:
>>>>>>>>>>>>> I think 50% could be reasonable to record also. Even if it's
>> a
>>>>>>> poor
>>>>>>>>>>> metric
>>>>>>>>>>>>> for operational purposes, a lot of people might expect to see
>>>>>>>>> "mean".
>>>>>>>>>>>>> Actually,
>>>>>>>>>>>>> I was surprised not to see "min". Is there a reason to leave
>> it
>>>>>>>>> off?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I might suggest:
>>>>>>>>>>>>> min, mean (50th), 75th, 99th, max
>>>>>>>>>>>>>
>>>>>>>>>>>>> B3:
>>>>>>>>>>>>> I agree we should include late records (though not the ones
>> we
>>>>>>>>> drop).
>>>>>>>>>>>>> It may be spiky, but only when there are legitimately some
>>>>>>> records
>>>>>>>>>>> with a
>>>>>>>>>>>>> high end-to-end latency, which is the whole point of these
>>>>>>> metrics.
>>>>>>>>>>>>>
>>>>>>>>>>>>> That's it! I don't think I have any other feedback, other
>> than
>>>>>>> a
>>>>>>>>>>> request to
>>>>>>>>>>>>> also report "min".
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> -John
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, May 13, 2020, at 16:58, Guozhang Wang wrote:
>>>>>>>>>>>>>> Thanks Sophie for the KIP, a few quick thoughts:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1) The end-to-end latency includes both the processing
>>>>>>> latency
>>>>>>>>> of the
>>>>>>>>>>>>> task
>>>>>>>>>>>>>> and the latency spent sitting in intermediate topics. I have
>>>>>>> a
>>>>>>>>>>> similar
>>>>>>>>>>>>>> feeling as Boyang mentioned above that the latency metric of
>>>>>>> a
>>>>>>>>> task A
>>>>>>>>>>>>>> actually measures the latency of the sub-topology up-to but
>>>>>>> not
>>>>>>>>>>> including
>>>>>>>>>>>>>> the processing of A, which is a bit weird.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Maybe the my feeling comes from the name "latency" itself,
>>>>>>> since
>>>>>>>>>>> today we
>>>>>>>>>>>>>> already have several "latency" metrics already which are
>>>>>>>>> measuring
>>>>>>>>>>>>> elapsed
>>>>>>>>>>>>>> system-time for processing a record / etc, while here we are
>>>>>>>>>>> comparing
>>>>>>>>>>>>> the
>>>>>>>>>>>>>> system wallclock time with the record timestamp.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Maybe we can consider renaming it as "record-staleness"
>>>>>>> (note we
>>>>>>>>>>> already
>>>>>>>>>>>>>> have a "record-lateness" metric), in which case recording at
>>>>>>> the
>>>>>>>>>>>>>> system-time before we start processing the record sounds
>> more
>>>>>>>>>>> natural.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2) With that in mind, I'm wondering if the
>>>>>>> processor-node-level
>>>>>>>>> DEBUG
>>>>>>>>>>>>>> metric is worth to add, given that we already have a
>>>>>>> task-level
>>>>>>>>>>>>> processing
>>>>>>>>>>>>>> latency metric. Basically, a specific node's e2e latency is
>>>>>>>>> similar
>>>>>>>>>>> to
>>>>>>>>>>>>> the
>>>>>>>>>>>>>> task-level e2e latency + task-level processing latency.
>>>>>>>>> Personally I
>>>>>>>>>>>>> think
>>>>>>>>>>>>>> having a task-level record-staleness metric is sufficient.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Guozhang
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, May 13, 2020 at 11:46 AM Sophie Blee-Goldman <
>>>>>>>>>>>>> sop...@confluent.io>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1. I felt that 50% was not a particularly useful gauge for
>>>>>>> this
>>>>>>>>>>>>> specific
>>>>>>>>>>>>>>> metric, as
>>>>>>>>>>>>>>> it's presumably most useful at putting an *upper *bound on
>>>>>>> the
>>>>>>>>>>> latency
>>>>>>>>>>>>> you
>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>> reasonably expect to see. I chose percentiles that would
>>>>>>>>> hopefully
>>>>>>>>>>>>> give a
>>>>>>>>>>>>>>> good
>>>>>>>>>>>>>>> sense of what *most* records will experience, and what
>>>>>>> *close
>>>>>>>>> to
>>>>>>>>>>> all*
>>>>>>>>>>>>>>> records
>>>>>>>>>>>>>>> will.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> However I'm not married to these specific numbers and
>>>>>>> could be
>>>>>>>>>>>>> convinced.
>>>>>>>>>>>>>>> Would be especially interested in hearing from users on
>>>>>>> this.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2. I'm inclined to not include the "hop-to-hop latency" in
>>>>>>>>> this KIP
>>>>>>>>>>>>> since
>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>> can always compute it themselves by subtracting the
>>>>>>> previous
>>>>>>>>> node's
>>>>>>>>>>>>>>> end-to-end latency. I guess we could do it either way since
>>>>>>>>> you can
>>>>>>>>>>>>> always
>>>>>>>>>>>>>>> compute one from the other, but I think the end-to-end
>>>>>>> latency
>>>>>>>>>>> feels
>>>>>>>>>>>>> more
>>>>>>>>>>>>>>> valuable as it's main motivation is not to debug
>>>>>>> bottlenecks
>>>>>>>>> in the
>>>>>>>>>>>>>>> topology but
>>>>>>>>>>>>>>> to give users a sense of how long it takes arecord to be
>>>>>>>>> reflected
>>>>>>>>>>> in
>>>>>>>>>>>>>>> certain parts
>>>>>>>>>>>>>>> of the topology. For example this might be useful for users
>>>>>>>>> who are
>>>>>>>>>>>>>>> wondering
>>>>>>>>>>>>>>> roughly when a record that was just produced will be
>>>>>>> included
>>>>>>>>> in
>>>>>>>>>>> their
>>>>>>>>>>>>> IQ
>>>>>>>>>>>>>>> results.
>>>>>>>>>>>>>>> Debugging is just a nice side effect -- but maybe I didn't
>>>>>>> make
>>>>>>>>>>> that
>>>>>>>>>>>>> clear
>>>>>>>>>>>>>>> enough
>>>>>>>>>>>>>>> in the KIP's motivation.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 3. Good question, I should address this in the KIP. The
>>>>>>> short
>>>>>>>>>>> answer is
>>>>>>>>>>>>>>> "yes",
>>>>>>>>>>>>>>> we will include late records. I added a paragraph to the
>>>>>>> end
>>>>>>>>> of the
>>>>>>>>>>>>>>> Proposed
>>>>>>>>>>>>>>> Changes section explaining the reasoning here, please let
>>>>>>> me
>>>>>>>>> know
>>>>>>>>>>> if
>>>>>>>>>>>>> you
>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>> any concerns.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 4. Assuming you're referring to the existing metric
>>>>>>>>>>> "process-latency",
>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> metric
>>>>>>>>>>>>>>> reflects the time for the literal Node#process method to
>>>>>>> run
>>>>>>>>>>> whereas
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> metric
>>>>>>>>>>>>>>> would always be measured relative to the event timestamp.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That said, the naming collision there is pretty confusing
>>>>>>> so
>>>>>>>>> I've
>>>>>>>>>>>>> renamed
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> metrics in this KIP to "end-to-end-latency" which I feel
>>>>>>> better
>>>>>>>>>>>>> reflects
>>>>>>>>>>>>>>> the nature
>>>>>>>>>>>>>>> of the metric anyway.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for the feedback!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, May 13, 2020 at 10:21 AM Boyang Chen <
>>>>>>>>>>>>> reluctanthero...@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for the KIP Sophie. Getting the E2E latency is
>>>>>>>>> important
>>>>>>>>>>> for
>>>>>>>>>>>>>>>> understanding the bottleneck of the application.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> A couple of questions and ideas:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1. Could you clarify the rational of picking 75, 99 and
>>>>>>> max
>>>>>>>>>>>>> percentiles?
>>>>>>>>>>>>>>>> Normally I see cases where we use 50, 90 percentile as
>>>>>>> well
>>>>>>>>> in
>>>>>>>>>>>>> production
>>>>>>>>>>>>>>>> systems.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2. The current latency being computed is cumulative, I.E
>>>>>>> if a
>>>>>>>>>>> record
>>>>>>>>>>>>> goes
>>>>>>>>>>>>>>>> through A -> B -> C, then P(C) = T(B->C) + P(B) =
>>>>>>> T(B->C) +
>>>>>>>>>>> T(A->B) +
>>>>>>>>>>>>>>> T(A)
>>>>>>>>>>>>>>>> and so on, where P() represents the captured latency,
>>>>>>> and T()
>>>>>>>>>>>>> represents
>>>>>>>>>>>>>>>> the time for transiting the records between two nodes,
>>>>>>>>> including
>>>>>>>>>>>>>>> processing
>>>>>>>>>>>>>>>> time. For monitoring purpose, maybe having T(B->C) and
>>>>>>>>> T(A->B)
>>>>>>>>>>> are
>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>> natural to view as "hop-to-hop latency", otherwise if
>>>>>>> there
>>>>>>>>> is a
>>>>>>>>>>>>> spike in
>>>>>>>>>>>>>>>> T(A->B), both P(B) and P(C) are affected in the same
>>>>>>> time.
>>>>>>>>> In the
>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>> spirit, the E2E latency is meaningful only when the
>>>>>>> record
>>>>>>>>> exits
>>>>>>>>>>>>> from the
>>>>>>>>>>>>>>>> sink as this marks the whole time this record spent
>>>>>>> inside
>>>>>>>>> the
>>>>>>>>>>>>> funnel. Do
>>>>>>>>>>>>>>>> you think we could have separate treatment for sink
>>>>>>> nodes and
>>>>>>>>>>> other
>>>>>>>>>>>>>>>> nodes, so that other nodes only count the time receiving
>>>>>>> the
>>>>>>>>>>> record
>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>> last hop? I'm not proposing a solution here, just want to
>>>>>>>>> discuss
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> alternative to see if it is reasonable.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 3. As we are going to monitor late arrival records as
>>>>>>> well,
>>>>>>>>> they
>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>> create some really spiky graphs when the out-of-order
>>>>>>>>> records are
>>>>>>>>>>>>>>>> interleaving with on time records. Should we also supply
>>>>>>> a
>>>>>>>>> smooth
>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>> of the latency metrics, or user should just take care of
>>>>>>> it
>>>>>>>>> by
>>>>>>>>>>>>> themself?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 4. Regarding this new metrics, we haven't discussed its
>>>>>>>>> relation
>>>>>>>>>>>>> with our
>>>>>>>>>>>>>>>> existing processing latency metrics, could you add some
>>>>>>>>> context
>>>>>>>>>>> on
>>>>>>>>>>>>>>>> comparison and a simple `when to use which` tutorial for
>>>>>>> the
>>>>>>>>>>> best?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Boyang
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, May 12, 2020 at 7:28 PM Sophie Blee-Goldman <
>>>>>>>>>>>>> sop...@confluent.io
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'd like to kick off discussion on KIP-613 which aims
>>>>>>> to
>>>>>>>>> add
>>>>>>>>>>>>> end-to-end
>>>>>>>>>>>>>>>>> latency metrics to Streams. Please take a look:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>> Sophie
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> -- Guozhang
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>>
>>>
>>
> 

Attachment: signature.asc
Description: OpenPGP digital signature

Reply via email to