[
https://issues.apache.org/jira/browse/BEAM-4775?focusedWorklogId=201652&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-201652
]
ASF GitHub Bot logged work on BEAM-4775:
----------------------------------------
Author: ASF GitHub Bot
Created on: 20/Feb/19 22:20
Start Date: 20/Feb/19 22:20
Worklog Time Spent: 10m
Work Description: ajamato commented on pull request #7876: [BEAM-4775]
Clean up metric protos; support integer distributions, gauges
URL: https://github.com/apache/beam/pull/7876#discussion_r258702084
##########
File path: model/fn-execution/src/main/proto/beam_fn_api.proto
##########
@@ -501,48 +501,9 @@ message MonitoringInfoTypeUrns {
message Metric {
// (Required) The data for this metric.
oneof data {
- CounterData counter_data = 1;
- DistributionData distribution_data = 2;
- ExtremaData extrema_data = 3;
- }
-}
-
-// Data associated with a Counter or Gauge metric.
-// This is designed to be compatible with metric collection
-// systems such as DropWizard.
-message CounterData {
- oneof value {
- int64 int64_value = 1;
- double double_value = 2;
- string string_value = 3;
- }
-}
-
-// Extrema messages are used for calculating
-// Top-N/Bottom-N metrics.
-message ExtremaData {
- oneof extrema {
- IntExtremaData int_extrema_data = 1;
- DoubleExtremaData double_extrema_data = 2;
- }
-}
-
-message IntExtremaData {
- repeated int64 int_values = 1;
-}
-
-message DoubleExtremaData {
- repeated double double_values = 2;
-}
-
-// Data associated with a distribution metric.
-// This is based off of the current DistributionData metric.
-// This is not a stackdriver or dropwizard compatible
-// style of distribution metric.
-message DistributionData {
- oneof distribution {
- IntDistributionData int_distribution_data = 1;
- DoubleDistributionData double_distribution_data = 2;
+ int64 counter = 1;
Review comment:
@robertwb I'd like to push back a bit about a significant change, where we
use opaque bytes payloads.
I am okay with removing the layers in this proto and using the design Ryan
has proposed here. But not an extensive rewrite at this stage of the game,
which will slow down progress significantly.
We have protos for the MonitoringInfoTable. This solution is the extensible
format, where you can store basically anything and the producer and consumer of
this proto can be the only ones who need to understand it (and it could be
passed through the RunnerHarness)
https://github.com/apache/beam/blob/197a06852d27c5baea6a4c65894a8841672af7b6/model/fn-execution/src/main/proto/beam_fn_api.proto#L583
But we chose to have separate protos for the common well known metric
formats (Metrics being a well defined concept shared by many systems, i.e. a
timeseries of data): counter, gauge, distribution, etc.
Additionally, if you wish to modify these formats significantly at a future
stage, you can always
- add to these protos without deleting
- Change the URN version number of a metric
- Upgrade all SDKs to package the metric with a format for the previous and
new version number of the URN
- Upgrade all Runners that wish to support it.
Upgrading the URN version is the safe way to do this without introducing a
breaking change, as consumers of MonitoringInfos can freely chose which
URNs+versions they support.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 201652)
Time Spent: 17h 50m (was: 17h 40m)
> JobService should support returning metrics
> -------------------------------------------
>
> Key: BEAM-4775
> URL: https://issues.apache.org/jira/browse/BEAM-4775
> Project: Beam
> Issue Type: Bug
> Components: beam-model
> Reporter: Eugene Kirpichov
> Assignee: Ryan Williams
> Priority: Major
> Labels: triaged
> Time Spent: 17h 50m
> Remaining Estimate: 0h
>
> [https://github.com/apache/beam/blob/master/model/job-management/src/main/proto/beam_job_api.proto]
> currently doesn't appear to have a way for JobService to return metrics to a
> user, even though
> [https://github.com/apache/beam/blob/master/model/fn-execution/src/main/proto/beam_fn_api.proto]
> includes support for reporting SDK metrics to the runner harness.
>
> Metrics are apparently necessary to run any ValidatesRunner tests because
> PAssert needs to validate that the assertions succeeded. However, this
> statement should be double-checked: perhaps it's possible to somehow work
> with PAssert without metrics support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
