Continuing what Rodric was saying I'd also like to bring into this picture Distributed Tracing from this older PR [1]. A holistic approach of "what to use, when, and for what purpose" would be useful.
[1] - https://github.com/apache/incubator-openwhisk/pull/2282 On Tue, Apr 17, 2018 at 2:59 PM Tyson Norris <[email protected]> wrote: > I took a brief look at the PR - it looks like “the prior approach” of > sending to couchdb is still enabled, is that correct? > > If so, it may be worthwhile to make the reference impl store to couchdb, > and remove the activation persistence from controller/invoker? > > This would also imply that the “polling” that is controller would also > need to be replaced? > Thanks > Tyson > > On Apr 17, 2018, at 12:02 PM, Rodric Rabbah <[email protected]<mailto: > [email protected]>> wrote: > > It would be useful to provide a reference implementation for consuming > this data. Can you also capture the goals in an issue (I scanned the PR > quickly but there’s no corresponding issue). > > Further now I think there are multiple ways of recording/reporting some of > the metrics (log markers which are largely silenced by default, kamon > metrics, and now kafka). Is that right? > > I think we’ll need to also document these and a guide for when to use > which - I caution that we are proliferating multiple ways of doing similar > things with no consistency or articulated long term vision. > > -r > > On Apr 17, 2018, at 12:01 PM, Vadim Raskin <[email protected]<mailto: > [email protected]>> wrote: > > Hi Chetan, > > Can you share some details on how this is currently being done with > CouchDB. Do we have any analytics view configured which computes these > numbers currently? > > To my knowledge we don't have the views that are shared anywhere in open > repos. > > May be we also include a basic default implementation out of the box > which collect aggregated stats using Kamon metrics already being used > > Not a bad idea, I'll consider sharing the peace of code after finishing the > development (separate from the original PR), it might require some > post-processing to strip away the ibm specific parts, so it might bide some > time. > > regards, > Vadim. > > > > On Tue, Apr 17, 2018 at 8:29 AM Chetan Mehrotra <[email protected] > <mailto:[email protected]>> > wrote: > > Hi Vadim, > > This looks helpful to get better insight in runtime operational stats! > > It has some advantages over to the prior approach (sending them to > CouchDB). > > Can you share some details on how this is currently being done with > CouchDB. Do we have any analytics view configured which computes these > numbers currently? > > Now it would be possible to simply connect a custom micro service > to Kafka and consume the activations in real-time. > > May be we also include a basic default implementation out of the box > which collect aggregated stats using Kamon metrics already being used > Chetan Mehrotra > > > On Mon, Apr 16, 2018 at 9:51 PM, Vadim Raskin <[email protected] > <mailto:[email protected]>> > wrote: > Hi everyone, > > > I’ve just opened a PR that enables sending activation metadata to Kafka. > It has some advantages over to the prior approach (sending them to > CouchDB). Now it would be possible to simply connect a custom micro > service > to Kafka and consume the activations in real-time. Some of the use cases > it > might cover: activation metrics - collect the data and push them into a > custom time-series database; user activity audit; activation analytics - > potentially get some insights with KSQL. > > > At the moment I’ve created a new kafka topic called events, which will > include messages from Controllers and Invokers. It encompasses the > following data collected from a single activation: > > > concurrentActivations > throttledActivations > statusCode > initTime > waitTime > duration > kind > > > Probably some more metadata will affiliate this list soon. > > > Just wanted to give a short heads up here. The PR I mentioned: > > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-openwhisk%2Fpull%2F3552&data=02%7C01%7Ctnorris%40adobe.com%7Cc908a394ad184dffaefe08d5a495c08c%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636595885445950512&sdata=seMpqJ1pUTYUt2a4XDUcBPTleDNCjxruf3HSjx%2BhDk4%3D&reserved=0 > > > Thank you, > > > Vadim. > >
