Hi everyone, Quick summary on python and Dataflow Runner: Python SDK already reports: - MSec - User metrics (int64 and distribution) - PCollection Element Count - Work on MeanByteCount for pcollection is ongoing here <https://github.com/apache/beam/pull/8062>.
Dataflow Runner: - all metrics listed above are passed through to Dataflow. Ryan can give more information on Flink Runner. I also see Maximilian on some of relevant PRs, so he might comment on this as well. Regards, Mikhail. On Thu, Apr 4, 2019 at 10:43 AM Pablo Estrada <pabl...@google.com> wrote: > Hello guys! > Alex, Mikhail and Ryan are working on support for metrics in the > portability framework. The support on the SDK is pretty advanced AFAIK*, > and the next step is to get the metrics back into the runner. Lukazs and > myself are working on a project that depends on this too, so I'm adding > everyone so we can get an idea of what's missing. > > I believe: > - User metrics are fully wired up in the SDK > - State sampler (timing) metrics are wired up as well (is that right, +Alex > Amato <ajam...@google.com>?) > - Work is ongoing to send the updates back to Flink. > - What is the plan for making metrics queriable from Flink? +Ryan Williams > <r...@runsascoded.com> > > Thanks! > -P. > > > > On Wed, Apr 3, 2019 at 12:02 PM Thomas Weise <t...@apache.org> wrote: > >> I believe this is where the metrics are supplied: >> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/operations.py >> >> git grep process_bundle_msecs yields results for dataflow worker only >> >> There isn't any test coverage for the Flink runner: >> >> >> https://github.com/apache/beam/blob/d38645ae8758d834c3e819b715a66dd82c78f6d4/sdks/python/apache_beam/runners/portability/flink_runner_test.py#L181 >> >> >> >> On Wed, Apr 3, 2019 at 10:45 AM Akshay Balwally <abalwa...@lyft.com> >> wrote: >> >>> Should have added- I'm using Python sdk, Flink runner >>> >>> On Wed, Apr 3, 2019 at 10:32 AM Akshay Balwally <abalwa...@lyft.com> >>> wrote: >>> >>>> Hi, >>>> I'm hoping to get metrics on the amount of time spent on each operator, >>>> so it seams like the stat >>>> >>>> >>>> {organization_specific_prefix}.operator.beam-metric-pardo_execution_time-process_bundle_msecs-v1.gauge.mean >>>> >>>> would be pretty helpful. But in practice, this stat always shows 0, >>>> which I interpret as 0 milliseconds spent per bundle, which can't be >>>> correct (other stats show that the operators are running, and timers within >>>> the operators show more reasonable times). Is this a known bug? >>>> >>>> >>>> -- >>>> *Akshay Balwally* >>>> Software Engineer >>>> 937.271.6469 <+19372716469> >>>> [image: Lyft] <http://www.lyft.com/> >>>> >>> >>> >>> -- >>> *Akshay Balwally* >>> Software Engineer >>> 937.271.6469 <+19372716469> >>> [image: Lyft] <http://www.lyft.com/> >>> >>
