[GitHub] [beam] robertwb commented on a change in pull request #11184: [WIP][BEAM-4374] Update protos related to MonitoringInfo.
robertwb commented on a change in pull request #11184: [WIP][BEAM-4374] Update
protos related to MonitoringInfo.
URL: https://github.com/apache/beam/pull/11184#discussion_r395935001
##
File path: model/pipeline/src/main/proto/metrics.proto
##
@@ -229,101 +215,127 @@ message MonitoringInfo {
NAMESPACE = 5 [(label_props) = { name: "NAMESPACE" }];
NAME = 6 [(label_props) = { name: "NAME" }];
}
+
// A set of key+value labels which define the scope of the metric.
// Either a well defined entity id for matching the enum names in
// the MonitoringInfoLabels enum or any arbitrary label
// set by a custom metric or user metric.
+ //
// A monitoring system is expected to be able to aggregate the metrics
// together for all updates having the same URN and labels. Some systems such
// as Stackdriver will be able to aggregate the metrics using a subset of the
// provided labels
- map labels = 5;
-
- // The walltime of the most recent update.
- // Useful for aggregation for latest types such as LatestInt64.
- google.protobuf.Timestamp timestamp = 6;
+ map labels = 4;
}
message MonitoringInfoTypeUrns {
enum Enum {
+// Represents an integer counter where values are summed across bundles.
+//
+// Encoding:
+// - value: beam:coder:varint:v1
SUM_INT64_TYPE = 0 [(org.apache.beam.model.pipeline.v1.beam_urn) =
Review comment:
Rather than manually write out the cross product, how about we define
`{sum,min,max,top_n,bottom_n,distribtuion,latest}_{int64,double,string}` types
as having known semantics and encoding?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] robertwb commented on a change in pull request #11184: [WIP][BEAM-4374] Update protos related to MonitoringInfo.
robertwb commented on a change in pull request #11184: [WIP][BEAM-4374] Update
protos related to MonitoringInfo.
URL: https://github.com/apache/beam/pull/11184#discussion_r395936249
##
File path: model/pipeline/src/main/proto/metrics.proto
##
@@ -52,38 +55,157 @@ message Annotation {
string value = 2;
}
-// Populated MonitoringInfoSpecs for specific URNs.
-// Indicating the required fields to be set.
-// SDKs and RunnerHarnesses can load these instances into memory and write a
-// validator or code generator to assist with populating and validating
-// MonitoringInfo protos.
+// A set of well known MonitoringInfo specifications.
message MonitoringInfoSpecs {
enum Enum {
-// TODO(BEAM-6926): Add the PTRANSFORM name as a required label after
-// upgrading the python SDK.
-USER_COUNTER = 0 [(monitoring_info_spec) = {
- urn: "beam:metric:user",
- type_urn: "beam:metrics:sum_int_64",
+// Represents an integer counter where values are summed across bundles.
+USER_SUM_INT64 = 0 [(monitoring_info_spec) = {
+ urn: "beam:metric:user:v1",
+ type: "beam:metrics:sum_int64:v1",
required_labels: ["PTRANSFORM", "NAMESPACE", "NAME"],
annotations: [{
key: "description",
-value: "URN utilized to report user numeric counters."
+value: "URN utilized to report user metric."
}]
}];
-ELEMENT_COUNT = 1 [(monitoring_info_spec) = {
+// Represents a double counter where values are summed across bundles.
+USER_SUM_DOUBLE = 1 [(monitoring_info_spec) = {
+ urn: "beam:metric:user:v1",
Review comment:
Should it be legal to have two counters with the same URN but different
types. (This seems to fly agains the idea of a URN being a Unique identifier.)
Seeing this explosion of types, however, makes it feel like we should not be
manually be enumerating them (or at least I'm struggling to see the value in
that over just saying that user counters may have any type).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] robertwb commented on a change in pull request #11184: [WIP][BEAM-4374] Update protos related to MonitoringInfo.
robertwb commented on a change in pull request #11184: [WIP][BEAM-4374] Update
protos related to MonitoringInfo.
URL: https://github.com/apache/beam/pull/11184#discussion_r395933544
##
File path: model/pipeline/src/main/proto/metrics.proto
##
@@ -229,121 +330,148 @@ message MonitoringInfo {
NAMESPACE = 5 [(label_props) = { name: "NAMESPACE" }];
NAME = 6 [(label_props) = { name: "NAME" }];
}
- // A set of key+value labels which define the scope of the metric.
+
+ // A set of key and value labels which define the scope of the metric. For
+ // well known URNs, the set of required labels is provided by the associated
+ // MonitoringInfoSpec.
+ //
// Either a well defined entity id for matching the enum names in
// the MonitoringInfoLabels enum or any arbitrary label
// set by a custom metric or user metric.
+ //
// A monitoring system is expected to be able to aggregate the metrics
// together for all updates having the same URN and labels. Some systems such
// as Stackdriver will be able to aggregate the metrics using a subset of the
// provided labels
- map labels = 5;
-
- // The walltime of the most recent update.
- // Useful for aggregation for latest types such as LatestInt64.
- google.protobuf.Timestamp timestamp = 6;
+ map labels = 4;
}
+// A set of well known URNs that specify the encoding and aggregation method.
message MonitoringInfoTypeUrns {
enum Enum {
+// Represents an integer counter where values are summed across bundles.
+//
+// Encoding:
+// - value: beam:coder:varint:v1
SUM_INT64_TYPE = 0 [(org.apache.beam.model.pipeline.v1.beam_urn) =
-"beam:metrics:sum_int_64"];
-
-DISTRIBUTION_INT64_TYPE = 1 [(org.apache.beam.model.pipeline.v1.beam_urn) =
- "beam:metrics:distribution_int_64"];
-
-LATEST_INT64_TYPE = 2 [(org.apache.beam.model.pipeline.v1.beam_urn) =
- "beam:metrics:latest_int_64"];
-
-// iterable is encoded with a beam:coder:double:v1 coder for each
-// element.
-LATEST_DOUBLES_TYPE = 3 [(org.apache.beam.model.pipeline.v1.beam_urn) =
- "beam:metrics:latest_doubles"];
- }
-}
-
-message Metric {
- // (Required) The data for this metric.
- oneof data {
-CounterData counter_data = 1;
-DistributionData distribution_data = 2;
-ExtremaData extrema_data = 3;
- }
-}
-
-// Data associated with a Counter or Gauge metric.
-// This is designed to be compatible with metric collection
-// systems such as DropWizard.
-message CounterData {
- oneof value {
-int64 int64_value = 1;
-double double_value = 2;
-string string_value = 3;
- }
-}
-
-// Extrema messages are used for calculating
-// Top-N/Bottom-N metrics.
-message ExtremaData {
- oneof extrema {
-IntExtremaData int_extrema_data = 1;
-DoubleExtremaData double_extrema_data = 2;
- }
-}
-
-message IntExtremaData {
- repeated int64 int_values = 1;
-}
+"beam:metrics:sum_int64:v1"];
+
+// Represents a double counter where values are summed across bundles.
+//
+// Encoding:
+// value: beam:coder:double:v1
+SUM_DOUBLE_TYPE = 1 [(org.apache.beam.model.pipeline.v1.beam_urn) =
+"beam:metrics:sum_double:v1"];
+
+// Represents a distribution of an integer value where:
+// - count: represents the number of values seen across all bundles
+// - sum: represents the total of the value across all bundles
+// - min: represents the smallest value seen across all bundles
+// - max: represents the largest value seen across all bundles
+//
+// Encoding:
+// - count: beam:coder:varint:v1
+// - sum: beam:coder:varint:v1
+// - min: beam:coder:varint:v1
+// - max: beam:coder:varint:v1
+DISTRIBUTION_INT64_TYPE = 2 [(org.apache.beam.model.pipeline.v1.beam_urn) =
+ "beam:metrics:distribution_int64:v1"];
+
+// Represents a distribution of a double value where:
+// - count: represents the number of values seen across all bundles
+// - sum: represents the total of the value across all bundles
+// - min: represents the smallest value seen across all bundles
+// - max: represents the largest value seen across all bundles
+//
+// Encoding:
+// - count: beam:coder:varint:v1
+// - sum: beam:coder:double:v1
+// - min: beam:coder:double:v1
+// - max: beam:coder:double:v1
+DISTRIBUTION_DOUBLE_TYPE = 3 [(org.apache.beam.model.pipeline.v1.beam_urn)
=
+ "beam:metrics:distribution_double:v1"];
+
+// Represents the latest seen integer value. The timestamp is used to
+// provide an "ordering" over multiple values to determine which is the
+// latest.
+//
+// Encoding:
+// - timestamp: beam:coder:varint:v1
[GitHub] [beam] robertwb commented on a change in pull request #11184: [WIP][BEAM-4374] Update protos related to MonitoringInfo.
robertwb commented on a change in pull request #11184: [WIP][BEAM-4374] Update
protos related to MonitoringInfo.
URL: https://github.com/apache/beam/pull/11184#discussion_r395933271
##
File path: model/pipeline/src/main/proto/metrics.proto
##
@@ -229,101 +215,127 @@ message MonitoringInfo {
NAMESPACE = 5 [(label_props) = { name: "NAMESPACE" }];
NAME = 6 [(label_props) = { name: "NAME" }];
}
+
// A set of key+value labels which define the scope of the metric.
// Either a well defined entity id for matching the enum names in
// the MonitoringInfoLabels enum or any arbitrary label
// set by a custom metric or user metric.
+ //
// A monitoring system is expected to be able to aggregate the metrics
// together for all updates having the same URN and labels. Some systems such
// as Stackdriver will be able to aggregate the metrics using a subset of the
// provided labels
- map labels = 5;
-
- // The walltime of the most recent update.
- // Useful for aggregation for latest types such as LatestInt64.
- google.protobuf.Timestamp timestamp = 6;
+ map labels = 4;
}
message MonitoringInfoTypeUrns {
enum Enum {
+// Represents an integer counter where values are summed across bundles.
+//
+// Encoding:
+// - value: beam:coder:varint:v1
SUM_INT64_TYPE = 0 [(org.apache.beam.model.pipeline.v1.beam_urn) =
-"beam:metrics:sum_int_64"];
-
-DISTRIBUTION_INT64_TYPE = 1 [(org.apache.beam.model.pipeline.v1.beam_urn) =
- "beam:metrics:distribution_int_64"];
-
-LATEST_INT64_TYPE = 2 [(org.apache.beam.model.pipeline.v1.beam_urn) =
- "beam:metrics:latest_int_64"];
+"beam:metrics:sum_int64:v1"];
+
+// Represents a double counter where values are summed across bundles.
+//
+// Encoding:
+// value: beam:coder:double:v1
+SUM_DOUBLE_TYPE = 1 [(org.apache.beam.model.pipeline.v1.beam_urn) =
+"beam:metrics:sum_int64:v1"];
+
+// Represents a distribution of an integer value where:
+// - count: represents the number of values seen across all bundles
+// - sum: represents the total of the value across all bundles
+// - min: represents the smallest value seen across all bundles
+// - max: represents the largest value seen across all bundles
+//
+// Encoding:
+// - count: beam:coder:varint:v1
+// - sum: beam:coder:varint:v1
+// - min: beam:coder:varint:v1
+// - max: beam:coder:varint:v1
+DISTRIBUTION_INT64_TYPE = 2 [(org.apache.beam.model.pipeline.v1.beam_urn) =
+ "beam:metrics:distribution_int64:v1"];
+
+// Represents a distribution of a double value where:
+// - count: represents the number of values seen across all bundles
+// - sum: represents the total of the value across all bundles
+// - min: represents the smallest value seen across all bundles
+// - max: represents the largest value seen across all bundles
+//
+// Encoding:
+// - count: beam:coder:varint:v1
+// - sum: beam:coder:double:v1
+// - min: beam:coder:double:v1
+// - max: beam:coder:double:v1
+DISTRIBUTION_DOUBLE_TYPE = 3 [(org.apache.beam.model.pipeline.v1.beam_urn)
=
+ "beam:metrics:distribution_int64:v1"];
+
+// Represents the latest seen integer value. The timestamp is used to
+// provide an "ordering" over multiple values to determine which is the
+// latest.
+//
+// Encoding:
+// - timestamp: beam:coder:varint:v1 (milliseconds since epoch)
+// - value: beam:coder:varint:v1
+LATEST_INT64_TYPE = 4 [(org.apache.beam.model.pipeline.v1.beam_urn) =
+ "beam:metrics:latest_int64:v1"];
+
+// Represents the latest seen integer value. The timestamp is used to
+// provide an "ordering" over multiple values to determine which is the
+// latest.
+//
+// Encoding:
+// - timestamp: beam:coder:varint:v1 (milliseconds since epoch)
+// - value: beam:coder:double:v1
+LATEST_DOUBLE_TYPE = 5 [(org.apache.beam.model.pipeline.v1.beam_urn) =
+ "beam:metrics:latest_int64:v1"];
+
+// Represents the largest set of integer values seen across bundles.
+//
+// Encoding: ...
+// - valueX: beam:coder:varint:v1
+TOP_N_INT64_TYPE = 6 [(org.apache.beam.model.pipeline.v1.beam_urn) =
+ "beam:metrics:top_n_int64:v1"];
+
+// Represents the largest set of double values seen across bundles.
+//
+// Encoding: ...
+// - valueX: beam:coder:double:v1
+TOP_N_DOUBLE_TYPE = 7 [(org.apache.beam.model.pipeline.v1.beam_urn) =
+"beam:metrics:top_n_int64:v1"];
+
+// Represents the smallest set of
