Nikita What is the difference between those two metrics?
ср, 24 июля 2019 г., 12:45 Nikita Amelchev <nsamelc...@gmail.com>: > Igniters, thanks for comments. > > From the discussion it can be seen that we need only two metrics for now: > - сacheOperationsBlockedDuration (long) > - totalCacheOperationsBlockedDuration (long) > > I will prepare PR at the nearest time. > > ср, 24 июл. 2019 г. в 09:11, Zhenya Stanilovsky <arzamas...@mail.ru.invalid > >: > > > > +1 with Anton decisions. > > > > > > >Среда, 24 июля 2019, 8:44 +03:00 от Anton Vinogradov <a...@apache.org>: > > > > > >Folks, > > > > > >It looks like we're trying to implement "extended debug" instead of > > >"monitoring". > > >It should not be interesting for real admin what phase of PME is in > > >progress and so on. > > >Interested metrics are > > >- total blocked time (will be used for real SLA counting) > > >- are we blocked right now (shows we have an SLA degradation right now) > > >Duration of the current blocking period can be easily presented using > any > > >modern monitoring tool by regular checks. > > >Initial true will means "period start", precision will be a result of > > >checks frequency. > > >Anyway, I'm ok to have current metric presented with long, where long > is a > > >duration, see no reason, but ok :) > > > > > >All other features you mentioned are useful for code or > > >deployment improving and can (should) be taken from logs at the analysis > > >phase. > > > > > >On Tue, Jul 23, 2019 at 7:22 PM Ivan Rakov < ivan.glu...@gmail.com > > wrote: > > > > > >> Folks, let me step in. > > >> > > >> Nikita, thanks for your suggestions! > > >> > > >> > 1. initialVersion. Topology version that initiates the exchange. > > >> > 2. initTime. Time PME was started. > > >> > 3. initEvent. Event that triggered PME. > > >> > 4. partitionReleaseTime. Time when a node has finished waiting for > all > > >> > updates and translations on a previous topology. > > >> > 5. sendSingleMessageTime. Time when a node sent a single message. > > >> > 6. recieveFullMessageTime. Time when a node received a full message. > > >> > 7. finishTime. Time PME was ended. > > >> > > > >> > When new PME started all these metrics resets. > > >> Every metric from Nikita's list looks useful and simple to implement. > > >> I think that it would be better to change format of metrics 4, 5, 6 > and > > >> 7 a bit: we can keep only difference between time of previous event > and > > >> time of corresponding event. Such metrics would be easier to perceive: > > >> they answer to specific questions "how much time did partition release > > >> take?" or "how much time did awaiting of distributed phase end take?". > > >> Also, if results of 4, 5, 6, 7 will be exported to monitoring system, > > >> graphs will show how different stages times change from one PME to > another. > > >> > > >> > When PME cause no blocking, it's a good PME and I see no reason to > have > > >> > monitoring related to it > > >> Agree with Anton here. These metrics should be measured only for true > > >> distributed exchange. Saving results for client leave/join PMEs will > > >> just complicate monitoring. > > >> > > >> > I agree with total blocking duration metric but > > >> > I still don't understand why instant value indicating that > operations are > > >> > blocked should be boolean. > > >> > Duration time since blocking has started looks more appropriate and > > >> useful. > > >> > It gives more information while semantic is left the same. > > >> Totally agree with Pavel here. Both "accumulated block time" and > > >> "current PME block time" metrics are useful. Growth of accumulated > > >> metric for specific period of time (should be easy to check via > > >> monitoring system graph) will show for how much business operations > were > > >> blocked in total, and non-zero current metric will show that we are > > >> experiencing issues right now. Boolean metric "are we blocked right > now" > > >> is not needed as it's obviously can be inferred from "current PME > block > > >> time". > > >> > > >> Best Regards, > > >> Ivan Rakov > > >> > > >> On 23.07.2019 16:02, Pavel Kovalenko wrote: > > >> > Nikita, > > >> > > > >> > I agree with total blocking duration metric but > > >> > I still don't understand why instant value indicating that > operations are > > >> > blocked should be boolean. > > >> > Duration time since blocking has started looks more appropriate and > > >> useful. > > >> > It gives more information while semantic is left the same. > > >> > > > >> > > > >> > > > >> > вт, 23 июл. 2019 г. в 11:42, Nikita Amelchev < nsamelc...@gmail.com > >: > > >> > > > >> >> Folks, > > >> >> > > >> >> All previous suggestions have some disadvantages. It can be several > > >> >> exchanges between two metric updates and fast exchange can rewrite > > >> >> previous long exchange. > > >> >> > > >> >> We can introduce a metric of total blocking duration that will > > >> >> accumulate at the end of the exchange. So, users will get actual > > >> >> information about how long operations were blocked. Cluster metric > > >> >> will be a maximum of local nodes metrics. And we need a boolean > metric > > >> >> that will indicate realtime status. It needs because of duration > > >> >> metric updates at the end of the exchange. > > >> >> > > >> >> So I propose to change the current metric that not released to the > > >> >> totalCacheOperationsBlockingDuration metric and to add the > > >> >> isCacheOperationsBlocked metric. > > >> >> > > >> >> WDYT? > > >> >> > > >> >> пн, 22 июл. 2019 г. в 09:27, Anton Vinogradov < a...@apache.org >: > > >> >>> Nikolay, > > >> >>> > > >> >>> Still see no reason to replace boolean with long. > > >> >>> > > >> >>> On Mon, Jul 22, 2019 at 9:19 AM Nikolay Izhikov < > nizhi...@apache.org > > > >> >> wrote: > > >> >>>> Anton. > > >> >>>> > > >> >>>> 1. Value exported based on SPI settings, not in the moment it > changed. > > >> >>>> > > >> >>>> 2. Clock synchronisation - if we export start time, we should > also > > >> >> export > > >> >>>> node local timestamp. > > >> >>>> > > >> >>>> пн, 22 июля 2019 г., 8:33 Anton Vinogradov < a...@apache.org >: > > >> >>>> > > >> >>>>> Folks, > > >> >>>>> > > >> >>>>> What's the reason for duration counting? > > >> >>>>> AFAIU, it's a monitoring system feature to count the durations. > > >> >>>>> Sine monitoring system checks metrics periodically it will know > the > > >> >>>>> duration by its own log. > > >> >>>>> > > >> >>>>> On Fri, Jul 19, 2019 at 7:32 PM Pavel Kovalenko < > jokse...@gmail.com > > > >> >>>>> wrote: > > >> >>>>> > > >> >>>>>> Nikita, > > >> >>>>>> > > >> >>>>>> Yes, I mean duration not timestamp. For the metric name, I > suggest > > >> >>>>>> "cacheOperationsBlockingDuration", I think it cleaner > represents > > >> >> what > > >> >>>> is > > >> >>>>>> blocked during PME. > > >> >>>>>> We can also combine both timestamp > > >> >> "cacheOperationsBlockingStartTs" and > > >> >>>>>> duration to have better correlation when cache operations were > > >> >> blocked > > >> >>>>> and > > >> >>>>>> how much time it's taken. > > >> >>>>>> For instant view (like in JMX bean) a calculated value as you > > >> >> mentioned > > >> >>>>>> can be used. > > >> >>>>>> For metrics are exported to some backend (IEP-35) a counter > can be > > >> >>>> used. > > >> >>>>>> The counter is incremented by blocking time after blocking has > > >> >> ended. > > >> >>>>>> пт, 19 июл. 2019 г. в 19:10, Nikita Amelchev < > nsamelc...@gmail.com > > >> >>> : > > >> >>>>>>> Pavel, > > >> >>>>>>> > > >> >>>>>>> The main purpose of this metric is > > >> >>>>>>>>> how much time we wait for resuming cache operations > > >> >>>>>>> Seems I misunderstood you. Do you mean timestamp or duration > here? > > >> >>>>>>>>> What do you think if we change the boolean value of metric > to a > > >> >>>> long > > >> >>>>>>> value that represents time in milliseconds when operations > were > > >> >>>> blocked? > > >> >>>>>>> This time can be calculated as (currentTime - > > >> >>>>>>> timeSinceOperationsBlocked) in case of timestamp. > > >> >>>>>>> > > >> >>>>>>> Duration will be more understandable. It'll be something like > > >> >>>>>>> getCurrentBlockingPmeDuration. But I haven't come up with a > better > > >> >>>>>>> name yet. > > >> >>>>>>> > > >> >>>>>>> пт, 19 июл. 2019 г. в 18:30, Pavel Kovalenko < > jokse...@gmail.com > > >> >>> : > > >> >>>>>>>> Nikita, > > >> >>>>>>>> > > >> >>>>>>>> I think getCurrentPmeDuration doesn't show useful > information. > > >> >> The > > >> >>>>> main > > >> >>>>>>> PME side effect for end-users is blocking cache operations. > Not > > >> >> all > > >> >>>> PME > > >> >>>>>>> time blocks it. > > >> >>>>>>>> What information gives to an end-user timestamp of > > >> >>>>>>> "timeSinceOperationsBlocked"? For what analysis it can be > used and > > >> >>>> how? > > >> >>>>>>>> пт, 19 июл. 2019 г. в 17:48, Nikita Amelchev < > > >> >> nsamelc...@gmail.com > > >> >>>>> : > > >> >>>>>>>>> Hi Pavel, > > >> >>>>>>>>> > > >> >>>>>>>>> This time already can be obtained from the > > >> >> getCurrentPmeDuration > > >> >>>> and > > >> >>>>>>>>> new isOperationsBlockedByPme metrics. > > >> >>>>>>>>> > > >> >>>>>>>>> As an alternative solution, I can rework recently added > > >> >>>>>>>>> getCurrentPmeDuration metric (not released yet). Seems for > > >> >> users it > > >> >>>>>>>>> useless in case of non-blocking PME. > > >> >>>>>>>>> Lets name it timeSinceOperationsBlocked. It'll be timestamp > > >> >> when > > >> >>>>>>>>> blocking started (minimal value of cluster nodes) and 0 if > > >> >> blocking > > >> >>>>>>>>> ends (there is no running PME). > > >> >>>>>>>>> > > >> >>>>>>>>> WDYT? > > >> >>>>>>>>> > > >> >>>>>>>>> пт, 19 июл. 2019 г. в 15:56, Pavel Kovalenko < > > >> >> jokse...@gmail.com >: > > >> >>>>>>>>>> Hi Nikita, > > >> >>>>>>>>>> > > >> >>>>>>>>>> Thank you for working on this. What do you think if we > > >> >> change the > > >> >>>>>>> boolean > > >> >>>>>>>>>> value of metric to a long value that represents time in > > >> >>>>> milliseconds > > >> >>>>>>> when > > >> >>>>>>>>>> operations were blocked? > > >> >>>>>>>>>> Since we have not only JMX and now metrics are periodically > > >> >>>>> exported > > >> >>>>>>> to > > >> >>>>>>>>>> some backend it can give a more clear picture of how much > > >> >> time we > > >> >>>>>>> wait for > > >> >>>>>>>>>> resuming cache operations instead of instant boolean > > >> >> indicator. > > >> >>>>>>>>>> пт, 19 июл. 2019 г. в 14:41, Nikita Amelchev < > > >> >>>> nsamelc...@gmail.com > > >> >>>>>> : > > >> >>>>>>>>>>> Anton, Nikolay, > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> Thanks for the support. > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> For now, we have the getCurrentPmeDuration() metric that > > >> >> does > > >> >>>> not > > >> >>>>>>> show > > >> >>>>>>>>>>> influence on the cluster correctly. PME can be without > > >> >> blocking > > >> >>>>>>>>>>> operations. For example, client node join/leave events. > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> I suggest add new metric - isOperationsBlockedByPme(). > > >> >>>> Together, > > >> >>>>>>> these > > >> >>>>>>>>>>> metrics will show influence of the PME on cluster and user > > >> >>>>>>> operations. > > >> >>>>>>>>>>> I have prepared PR for this (Bot visa is green). [1] Can > > >> >> anyone > > >> >>>>>>> take a > > >> >>>>>>>>>>> look? > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-11961 > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> вт, 16 июл. 2019 г. в 14:58, Nikolay Izhikov < > > >> >>>>> nizhi...@apache.org > > >> >>>>>>>> : > > >> >>>>>>>>>>>> I think administator of Ignite cluster should be able to > > >> >>>>> monitor > > >> >>>>>>> all > > >> >>>>>>>>>>> Ignite process, including non blocking PME. > > >> >>>>>>>>>>>> В Вт, 16/07/2019 в 14:57 +0300, Anton Vinogradov пишет: > > >> >>>>>>>>>>>>> BTW, > > >> >>>>>>>>>>>>> Found PME metric - getCurrentPmeDuration(). > > >> >>>>>>>>>>>>> Seems, it shows exactly PME time and not so useful > > >> >> because > > >> >>>> of > > >> >>>>>>> this. > > >> >>>>>>>>>>>>> The goal it so show exactly blocking period. > > >> >>>>>>>>>>>>> When PME cause no blocking, it's a good PME and I see > > >> >> no > > >> >>>>>>> reason to have > > >> >>>>>>>>>>>>> monitoring related to it :) > > >> >>>>>>>>>>>>> > > >> >>>>>>>>>>>>> On Tue, Jul 16, 2019 at 2:50 PM Nikolay Izhikov < > > >> >>>>>>> nizhi...@apache.org > > > >> >>>>>>>>>>> wrote: > > >> >>>>>>>>>>>>>> Anton. > > >> >>>>>>>>>>>>>> > > >> >>>>>>>>>>>>>> Why do we need to postpone implementation of this > > >> >>>> metrics? > > >> >>>>>>>>>>>>>> For now, implementation of new metric is very simple. > > >> >>>>>>>>>>>>>> > > >> >>>>>>>>>>>>>> I think we can implement this metrics as a single > > >> >>>>>>> contribution. > > >> >>>>>>>>>>>>>> В Вт, 16/07/2019 в 13:47 +0300, Anton Vinogradov > > >> >> пишет: > > >> >>>>>>>>>>>>>>> Nikita, > > >> >>>>>>>>>>>>>>> > > >> >>>>>>>>>>>>>>> Looks like all we need now is a 1 simple metric: > > >> >> are > > >> >>>>>>> operations > > >> >>>>>>>>>>> blocked? > > >> >>>>>>>>>>>>>>> Just a true or false. > > >> >>>>>>>>>>>>>>> Lest start from this. > > >> >>>>>>>>>>>>>>> All other metrics can be extracted from logs now > > >> >> and > > >> >>>> can > > >> >>>>> be > > >> >>>>>>>>>>> implemented > > >> >>>>>>>>>>>>>>> later. > > >> >>>>>>>>>>>>>>> > > >> >>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 12:46 PM Nikolay Izhikov < > > >> >>>>>>>>>>> nizhi...@apache.org > > > >> >>>>>>>>>>>>>>> wrote: > > >> >>>>>>>>>>>>>>> > > >> >>>>>>>>>>>>>>>> +1. > > >> >>>>>>>>>>>>>>>> > > >> >>>>>>>>>>>>>>>> Nikita, please, go ahead. > > >> >>>>>>>>>>>>>>>> > > >> >>>>>>>>>>>>>>>> > > >> >>>>>>>>>>>>>>>> вт, 16 июля 2019 г., 11:45 Nikita Amelchev < > > >> >>>>>>> nsamelc...@gmail.com > > >> >>>>>>>>>>>> : > > >> >>>>>>>>>>>>>>>>> Hello, Igniters. > > >> >>>>>>>>>>>>>>>>> > > >> >>>>>>>>>>>>>>>>> I suggest to add some useful metrics about the > > >> >>>>>>> partition map > > >> >>>>>>>>>>> exchange > > >> >>>>>>>>>>>>>>>>> (PME). For now, the duration of PME stages > > >> >>>> available > > >> >>>>>>> only in > > >> >>>>>>>>>>> log > > >> >>>>>>>>>>>>>> files > > >> >>>>>>>>>>>>>>>>> and cannot be obtained using JMX or other > > >> >> external > > >> >>>>>>> tools. [1] > > >> >>>>>>>>>>>>>>>>> I made the list of local node metrics that > > >> >> help to > > >> >>>>>>> understand > > >> >>>>>>>>>>> the > > >> >>>>>>>>>>>>>>>>> actual status of current PME: > > >> >>>>>>>>>>>>>>>>> > > >> >>>>>>>>>>>>>>>>> 1. initialVersion. Topology version that > > >> >> initiates > > >> >>>>> the > > >> >>>>>>>>>>> exchange. > > >> >>>>>>>>>>>>>>>>> 2. initTime. Time PME was started. > > >> >>>>>>>>>>>>>>>>> 3. initEvent. Event that triggered PME. > > >> >>>>>>>>>>>>>>>>> 4. partitionReleaseTime. Time when a node has > > >> >>>>> finished > > >> >>>>>>> waiting > > >> >>>>>>>>>>> for > > >> >>>>>>>>>>>>>> all > > >> >>>>>>>>>>>>>>>>> updates and translations on a previous > > >> >> topology. > > >> >>>>>>>>>>>>>>>>> 5. sendSingleMessageTime. Time when a node > > >> >> sent a > > >> >>>>>>> single > > >> >>>>>>>>>>> message. > > >> >>>>>>>>>>>>>>>>> 6. recieveFullMessageTime. Time when a node > > >> >>>> received > > >> >>>>> a > > >> >>>>>>> full > > >> >>>>>>>>>>> message. > > >> >>>>>>>>>>>>>>>>> 7. finishTime. Time PME was ended. > > >> >>>>>>>>>>>>>>>>> > > >> >>>>>>>>>>>>>>>>> When new PME started all these metrics resets. > > >> >>>>>>>>>>>>>>>>> > > >> >>>>>>>>>>>>>>>>> These metrics help to understand: > > >> >>>>>>>>>>>>>>>>> - how long PME was (current or previous). > > >> >>>>>>>>>>>>>>>>> - how long awaited for all updates was > > >> >> completed. > > >> >>>>>>>>>>>>>>>>> - what node blocks PME (didn't send a single > > >> >>>> message) > > >> >>>>>>>>>>>>>>>>> - what triggered PME. > > >> >>>>>>>>>>>>>>>>> > > >> >>>>>>>>>>>>>>>>> Thoughts? > > >> >>>>>>>>>>>>>>>>> > > >> >>>>>>>>>>>>>>>>> [1] > > >> >>>>> https://issues.apache.org/jira/browse/IGNITE-11961 > > >> >>>>>>>>>>>>>>>>> -- > > >> >>>>>>>>>>>>>>>>> Best wishes, > > >> >>>>>>>>>>>>>>>>> Amelchev Nikita > > >> >>>>>>>>>>>>>>>>> > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> -- > > >> >>>>>>>>>>> Best wishes, > > >> >>>>>>>>>>> Amelchev Nikita > > >> >>>>>>>>>>> > > >> >>>>>>>>> > > >> >>>>>>>>> > > >> >>>>>>>>> -- > > >> >>>>>>>>> Best wishes, > > >> >>>>>>>>> Amelchev Nikita > > >> >>>>>>> > > >> >>>>>>> > > >> >>>>>>> -- > > >> >>>>>>> Best wishes, > > >> >>>>>>> Amelchev Nikita > > >> >>>>>>> > > >> >> > > >> >> > > >> >> -- > > >> >> Best wishes, > > >> >> Amelchev Nikita > > >> >> > > >> > > > > > > -- > > Zhenya Stanilovsky > > > > -- > Best wishes, > Amelchev Nikita >
