Thanks Chesnay, Just for completeness, are there any relevant tickets for the discussion that one can follow, upvote, contribute to?
M On Tue, Aug 29, 2017 at 8:57 PM, Chesnay Schepler <ches...@apache.org> wrote: > Hello, > > 1. Because no one found time to fix it. In contrast to the remaining > byte/record metrics, input metrics for sources / output metrics for sinks > have to be implemented for every single implementation with their > respective semantics. In contrast, the output metrics are gathered in the > intersection between operators, independent of the actual operator > implementation. Furthermore, this requires system metrics (i.e. metrics > that Flink itself creates) to be exposed (and be mutable!) to user-defined > functions, which is something i *generally *wanted to avoid, but it > appears to be a big enough pain point to make an exception here. > > 2. Due to the above it is currently not possible without modifications of > the code to know how many reads/writes were made. > > 3. Do you mean aggregated metrics? The web UI allows the aggregation of > record/byte metrics on the task level. Beyond that we defer aggregation to > actual time-series databases that specialize in these things. > > > On 28.08.2017 19:08, Martin Eden wrote: > > Hi all, > > Just 3 quick questions both related to Flink metrics, especially around > sinks: > > 1. In the Flink UI Sources always have 0 input records / bytes and Sinks > always have 0 output records / bytes? Why is it like that? > > 2. What is the best practice for instrumenting off the shelf Flink sinks? > > Currently the only metrics available are num records/bytes in and out at > the operator and task scope. For the task scope there are extra buffer > metrics. However the output metrics are always zero (see question 1). How > can one know the actual number of successful writes done by an off the > shelf Flink sink? Or the latency of the write operation? > > 3. Is it possible to configure Flink to get global job metrics for all > subtasks of an operator? Or are there any best practices around that? > > Thanks, > M > > >
