[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=189028&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-189028 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 23/Jan/19 16:04 Start Date: 23/Jan/19 16:04 Worklog Time Spent: 10m Work Description: swegner commented on issue #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#issuecomment-456859715 The Java failure is a known-issue failing on master. I believe this is ok to merge. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 189028) Time Spent: 14.5h (was: 14h 20m) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 14.5h > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=189029&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-189029 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 23/Jan/19 16:04 Start Date: 23/Jan/19 16:04 Worklog Time Spent: 10m Work Description: swegner commented on pull request #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 189029) Time Spent: 14h 40m (was: 14.5h) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 14h 40m > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=188524&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-188524 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 22/Jan/19 22:43 Start Date: 22/Jan/19 22:43 Worklog Time Spent: 10m Work Description: Ardagan commented on issue #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#issuecomment-456592205 Update on failures: I ran ./gradlew :beam-runners-direct-java:needsRunnerTests locally and it passes. Seem to be flake. Looking on how can I run python tests. Running :pythonPreCommit fails installing docs deps. Checking how can I repro the error. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 188524) Time Spent: 14h 10m (was: 14h) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 14h 10m > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=188525&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-188525 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 22/Jan/19 22:45 Start Date: 22/Jan/19 22:45 Worklog Time Spent: 10m Work Description: Ardagan commented on issue #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#issuecomment-456592635 Rebased on master. See if this was some merge issue. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 188525) Time Spent: 14h 20m (was: 14h 10m) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 14h 20m > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=188515&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-188515 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 22/Jan/19 22:29 Start Date: 22/Jan/19 22:29 Worklog Time Spent: 10m Work Description: Ardagan commented on issue #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#issuecomment-456583437 run python precommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 188515) Time Spent: 13h 50m (was: 13h 40m) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 13h 50m > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=188516&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-188516 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 22/Jan/19 22:29 Start Date: 22/Jan/19 22:29 Worklog Time Spent: 10m Work Description: Ardagan commented on issue #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#issuecomment-456583397 run java precommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 188516) Time Spent: 14h (was: 13h 50m) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 14h > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=188514&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-188514 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 22/Jan/19 22:29 Start Date: 22/Jan/19 22:29 Worklog Time Spent: 10m Work Description: Ardagan commented on issue #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#issuecomment-456588179 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 188514) Time Spent: 13h 40m (was: 13.5h) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 13h 40m > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=188506&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-188506 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 22/Jan/19 22:12 Start Date: 22/Jan/19 22:12 Worklog Time Spent: 10m Work Description: Ardagan commented on issue #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#issuecomment-456583397 run java precommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 188506) Time Spent: 13h 20m (was: 13h 10m) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 13h 20m > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=188507&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-188507 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 22/Jan/19 22:12 Start Date: 22/Jan/19 22:12 Worklog Time Spent: 10m Work Description: Ardagan commented on issue #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#issuecomment-456583437 run python precommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 188507) Time Spent: 13.5h (was: 13h 20m) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 13.5h > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=188477&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-188477 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 22/Jan/19 20:41 Start Date: 22/Jan/19 20:41 Worklog Time Spent: 10m Work Description: swegner commented on issue #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#issuecomment-456555197 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 188477) Time Spent: 13h 10m (was: 13h) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 13h 10m > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=186488&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-186488 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 17/Jan/19 18:48 Start Date: 17/Jan/19 18:48 Worklog Time Spent: 10m Work Description: Ardagan commented on issue #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#issuecomment-455285133 Rebased on top of master. +committer: @swegner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 186488) Time Spent: 13h (was: 12h 50m) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 13h > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=186481&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-186481
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 17/Jan/19 18:23
Start Date: 17/Jan/19 18:23
Worklog Time Spent: 10m
Work Description: Ardagan commented on pull request #7323: [BEAM-6181]
Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r248786476
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/UserMonitoringInfoToCounterUpdateTransformer.java
##
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.fn.control;
+
+import com.google.api.services.dataflow.model.CounterMetadata;
+import com.google.api.services.dataflow.model.CounterStructuredName;
+import com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata;
+import com.google.api.services.dataflow.model.CounterUpdate;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfoSpecs.Enum;
+import org.apache.beam.runners.core.metrics.SpecMonitoringInfoValidator;
+import
org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowStepContext;
+import
org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Origin;
+import
org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Class for transforming MonitoringInfo's containing User counter values, to
relevant CounterUpdate
+ * proto.
+ */
+class UserMonitoringInfoToCounterUpdateTransformer
+implements MonitoringInfoToCounterUpdateTransformer {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(BeamFnMapTaskExecutor.class);
+
+ private final Map transformIdMapping;
+
+ private final SpecMonitoringInfoValidator specValidator;
+
+ public UserMonitoringInfoToCounterUpdateTransformer(
+ final SpecMonitoringInfoValidator specMonitoringInfoValidator,
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+this.specValidator = specMonitoringInfoValidator;
+ }
+
+ static final String BEAM_METRICS_USER_PREFIX =
Review comment:
Synced offline. BeamUrns extracts value from MonitoringInfoUrns, while
UserMonitoringInfoToCounterUpdateTransformer utilizes MonitoringInfoSpecs.
Decided to keep this PR as-is and have a separate PR to remove
MonitoringInfoUrns enum and utilize MonitoringInfoSpecs as source of truth.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 186481)
Time Spent: 12h 40m (was: 12.5h)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 12h 40m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=186482&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-186482
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 17/Jan/19 18:23
Start Date: 17/Jan/19 18:23
Worklog Time Spent: 10m
Work Description: Ardagan commented on pull request #7323: [BEAM-6181]
Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r248786646
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/UserMonitoringInfoToCounterUpdateTransformer.java
##
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.fn.control;
+
+import com.google.api.services.dataflow.model.CounterMetadata;
+import com.google.api.services.dataflow.model.CounterStructuredName;
+import com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata;
+import com.google.api.services.dataflow.model.CounterUpdate;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfoSpecs.Enum;
+import org.apache.beam.runners.core.metrics.SpecMonitoringInfoValidator;
+import
org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowStepContext;
+import
org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Origin;
+import
org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Class for transforming MonitoringInfo's containing User counter values, to
relevant CounterUpdate
+ * proto.
+ */
+class UserMonitoringInfoToCounterUpdateTransformer
+implements MonitoringInfoToCounterUpdateTransformer {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(BeamFnMapTaskExecutor.class);
+
+ private final Map transformIdMapping;
+
+ private final SpecMonitoringInfoValidator specValidator;
+
+ public UserMonitoringInfoToCounterUpdateTransformer(
+ final SpecMonitoringInfoValidator specMonitoringInfoValidator,
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+this.specValidator = specMonitoringInfoValidator;
+ }
+
+ static final String BEAM_METRICS_USER_PREFIX =
+ Enum.USER_COUNTER
+ .getValueDescriptor()
+ .getOptions()
+ .getExtension(BeamFnApi.monitoringInfoSpec)
+ .getUrn();
+
+ private Optional validate(MonitoringInfo monitoringInfo) {
+Optional validatorResult = specValidator.validate(monitoringInfo);
+if (validatorResult.isPresent()) {
+ return validatorResult;
+}
+
+String urn = monitoringInfo.getUrn();
+if (!urn.startsWith(BEAM_METRICS_USER_PREFIX)) {
+ throw new RuntimeException(
+ String.format(
+ "Received unexpected counter urn. Expected urn starting with:
%s, received: %s",
+ BEAM_METRICS_USER_PREFIX, urn));
+}
+
+final String ptransform = monitoringInfo.getLabelsMap().get("PTRANSFORM");
+DataflowStepContext stepContext = transformIdMapping.get(ptransform);
+if (stepContext == null) {
+ return Optional.of(
+ "Encountered user-counter MonitoringInfo with unknown ptransformId: "
+ + monitoringInfo.toString());
+}
+return Optional.empty();
+ }
+
+ /**
+ * Transforms user counter MonitoringInfo to relevant CounterUpdate.
+ *
+ * @return Relevant CounterUpdate or null if transformation failed.
+ */
+ @Override
+ public CounterUpdate transform(MonitoringInfo monitoringInfo) {
+Optional validationResult = validate(monitoringInfo);
+if (validationResult.isPresent()) {
+ LOG.info(validationResult.get());
+ return null;
+}
+
+long value = monitoringInfo.getMetric().getCounterData().getInt64Value();
+String urn = monito
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=183965&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-183965 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 10/Jan/19 22:08 Start Date: 10/Jan/19 22:08 Worklog Time Spent: 10m Work Description: Ardagan commented on issue #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#issuecomment-453273147 run java precommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 183965) Time Spent: 12.5h (was: 12h 20m) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 12.5h > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=183964&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-183964 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 10/Jan/19 22:07 Start Date: 10/Jan/19 22:07 Worklog Time Spent: 10m Work Description: Ardagan commented on issue #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#issuecomment-453273147 run java precommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 183964) Time Spent: 12h 20m (was: 12h 10m) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 12h 20m > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=183428&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-183428
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 10/Jan/19 00:27
Start Date: 10/Jan/19 00:27
Worklog Time Spent: 10m
Work Description: ajamato commented on pull request #7323: [BEAM-6181]
Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r246598604
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/UserMonitoringInfoToCounterUpdateTransformer.java
##
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.fn.control;
+
+import com.google.api.services.dataflow.model.CounterMetadata;
+import com.google.api.services.dataflow.model.CounterStructuredName;
+import com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata;
+import com.google.api.services.dataflow.model.CounterUpdate;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfoSpecs.Enum;
+import org.apache.beam.runners.core.metrics.SpecMonitoringInfoValidator;
+import
org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowStepContext;
+import
org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Origin;
+import
org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Class for transforming MonitoringInfo's containing User counter values, to
relevant CounterUpdate
+ * proto.
+ */
+class UserMonitoringInfoToCounterUpdateTransformer
+implements MonitoringInfoToCounterUpdateTransformer {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(BeamFnMapTaskExecutor.class);
+
+ private final Map transformIdMapping;
+
+ private final SpecMonitoringInfoValidator specValidator;
+
+ public UserMonitoringInfoToCounterUpdateTransformer(
+ final SpecMonitoringInfoValidator specMonitoringInfoValidator,
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+this.specValidator = specMonitoringInfoValidator;
+ }
+
+ static final String BEAM_METRICS_USER_PREFIX =
+ Enum.USER_COUNTER
+ .getValueDescriptor()
+ .getOptions()
+ .getExtension(BeamFnApi.monitoringInfoSpec)
+ .getUrn();
+
+ private Optional validate(MonitoringInfo monitoringInfo) {
+Optional validatorResult = specValidator.validate(monitoringInfo);
+if (validatorResult.isPresent()) {
+ return validatorResult;
+}
+
+String urn = monitoringInfo.getUrn();
+if (!urn.startsWith(BEAM_METRICS_USER_PREFIX)) {
+ throw new RuntimeException(
+ String.format(
+ "Received unexpected counter urn. Expected urn starting with:
%s, received: %s",
+ BEAM_METRICS_USER_PREFIX, urn));
+}
+
+final String ptransform = monitoringInfo.getLabelsMap().get("PTRANSFORM");
+DataflowStepContext stepContext = transformIdMapping.get(ptransform);
+if (stepContext == null) {
+ return Optional.of(
+ "Encountered user-counter MonitoringInfo with unknown ptransformId: "
+ + monitoringInfo.toString());
+}
+return Optional.empty();
+ }
+
+ /**
+ * Transforms user counter MonitoringInfo to relevant CounterUpdate.
+ *
+ * @return Relevant CounterUpdate or null if transformation failed.
+ */
+ @Override
+ public CounterUpdate transform(MonitoringInfo monitoringInfo) {
+Optional validationResult = validate(monitoringInfo);
+if (validationResult.isPresent()) {
+ LOG.info(validationResult.get());
+ return null;
+}
+
+long value = monitoringInfo.getMetric().getCounterData().getInt64Value();
+String urn = monito
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=183431&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-183431
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 10/Jan/19 00:27
Start Date: 10/Jan/19 00:27
Worklog Time Spent: 10m
Work Description: ajamato commented on pull request #7323: [BEAM-6181]
Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r246599753
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -589,6 +504,8 @@ public void stop() {
// Set final metrics to precisely the values in this update. This should
overwrite, not
// be combined with, all prior updates.
+ counterUpdates.clear();
+
Review comment:
rm blank line
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 183431)
Time Spent: 12h 10m (was: 12h)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 12h 10m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=183427&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-183427
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 10/Jan/19 00:27
Start Date: 10/Jan/19 00:27
Worklog Time Spent: 10m
Work Description: ajamato commented on pull request #7323: [BEAM-6181]
Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r246599649
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkItemStatusClient.java
##
@@ -266,27 +267,42 @@ private synchronized WorkItemStatus
createStatusUpdate(boolean isFinal) {
return status;
}
- // todo(migryz) this method should return List instead of
updating member variable
@VisibleForTesting
synchronized void populateCounterUpdates(WorkItemStatus status) {
if (worker == null) {
return;
}
+// Checking against boolean, because getCompleted can return null
boolean isFinalUpdate = Boolean.TRUE.equals(status.getCompleted());
-ImmutableList.Builder counterUpdatesListBuilder =
ImmutableList.builder();
+Map counterUpdatesMap = new HashMap<>();
+
+final Consumer appendCounterUpdate =
+x ->
+counterUpdatesMap.put(
+x.getStructuredNameAndMetadata() == null
+? x.getNameAndKind()
+: x.getStructuredNameAndMetadata(),
+x);
+
// Output counters
-
counterUpdatesListBuilder.addAll(extractCounters(worker.getOutputCounters()));
+extractCounters(worker.getOutputCounters()).forEach(appendCounterUpdate);
+
// User metrics reported in Worker
-counterUpdatesListBuilder.addAll(extractMetrics(isFinalUpdate));
+extractMetrics(isFinalUpdate).forEach(appendCounterUpdate);
+
+// counterUpdatesListBuilder.addAll(extractMetrics(isFinalUpdate));
Review comment:
Please delete commented out code before checking in
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 183427)
Time Spent: 11.5h (was: 11h 20m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 11.5h
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=183426&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-183426
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 10/Jan/19 00:27
Start Date: 10/Jan/19 00:27
Worklog Time Spent: 10m
Work Description: ajamato commented on pull request #7323: [BEAM-6181]
Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r246595742
##
File path:
runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/SpecMonitoringInfoValidatorTest.java
##
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.metrics;
+
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo;
+import org.junit.Before;
+import org.junit.Test;
+
+/** Relevant tests. */
+public class SpecMonitoringInfoValidatorTest {
+
+ SpecMonitoringInfoValidator testObject = null;
+
+ @Before
+ public void setUp() throws Exception {
+testObject = new SpecMonitoringInfoValidator();
+ }
+
+ @Test
+ public void validateReturnsErrorOnInvalidMonitoringInfoType() {
+MonitoringInfo testInput =
+MonitoringInfo.newBuilder()
+.setUrn("beam:metric:user:someCounter")
+.setType("beam:metrics:bad_value")
+.build();
+assertTrue(testObject.validate(testInput).isPresent());
+ }
+
+ @Test
+ public void validateReturnsNoErrorOnValidMonitoringInfo() {
+MonitoringInfo testInput =
+MonitoringInfo.newBuilder()
+.setUrn("beam:metric:user:someCounter")
+.setType("beam:metrics:sum_int_64")
+.putLabels("dummy", "value")
+.build();
+assertFalse(testObject.validate(testInput).isPresent());
+
+testInput =
+MonitoringInfo.newBuilder()
+.setUrn("beam:metric:element_count:v1")
Review comment:
I would like the shared constants to be used as much has possible. So there
is a single place where it is defined, incase it needs to be chagned.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 183426)
Time Spent: 11h 20m (was: 11h 10m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 11h 20m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=183429&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-183429
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 10/Jan/19 00:27
Start Date: 10/Jan/19 00:27
Worklog Time Spent: 10m
Work Description: ajamato commented on pull request #7323: [BEAM-6181]
Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r246596413
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/UserMonitoringInfoToCounterUpdateTransformer.java
##
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.fn.control;
+
+import com.google.api.services.dataflow.model.CounterMetadata;
+import com.google.api.services.dataflow.model.CounterStructuredName;
+import com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata;
+import com.google.api.services.dataflow.model.CounterUpdate;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfoSpecs.Enum;
+import org.apache.beam.runners.core.metrics.SpecMonitoringInfoValidator;
+import
org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowStepContext;
+import
org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Origin;
+import
org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Class for transforming MonitoringInfo's containing User counter values, to
relevant CounterUpdate
+ * proto.
+ */
+class UserMonitoringInfoToCounterUpdateTransformer
+implements MonitoringInfoToCounterUpdateTransformer {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(BeamFnMapTaskExecutor.class);
+
+ private final Map transformIdMapping;
+
+ private final SpecMonitoringInfoValidator specValidator;
+
+ public UserMonitoringInfoToCounterUpdateTransformer(
+ final SpecMonitoringInfoValidator specMonitoringInfoValidator,
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+this.specValidator = specMonitoringInfoValidator;
+ }
+
+ static final String BEAM_METRICS_USER_PREFIX =
Review comment:
Then can you use a shared static helper method to access it, so we don't
duplicate the 4 getter calls below.
i.e. something like.
static final String BEAM_METRICS_USER_PREFIX =
SimpleMonitoringInfo,getUrn(Enum.USER_COUNTER);
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 183429)
Time Spent: 11h 50m (was: 11h 40m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 11h 50m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=183430&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-183430 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 10/Jan/19 00:27 Start Date: 10/Jan/19 00:27 Worklog Time Spent: 10m Work Description: ajamato commented on pull request #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#discussion_r246596772 ## File path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/FnApiMonitoringInfoToCounterUpdateTransformer.java ## @@ -0,0 +1,66 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.beam.runners.dataflow.worker.fn.control; + +import com.google.api.services.dataflow.model.CounterUpdate; +import com.google.common.annotations.VisibleForTesting; +import java.util.HashMap; +import java.util.Map; +import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo; +import org.apache.beam.runners.core.metrics.SpecMonitoringInfoValidator; +import org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowStepContext; + +public class FnApiMonitoringInfoToCounterUpdateTransformer Review comment: Please add javadoc comments to the class, ctors and public methods This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 183430) Time Spent: 12h (was: 11h 50m) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 12h > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=178740&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-178740
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 26/Dec/18 18:54
Start Date: 26/Dec/18 18:54
Worklog Time Spent: 10m
Work Description: Ardagan commented on pull request #7323: [BEAM-6181]
Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r244034873
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkItemStatusClient.java
##
@@ -266,27 +266,62 @@ private synchronized WorkItemStatus
createStatusUpdate(boolean isFinal) {
return status;
}
- // todo(migryz) this method should return List instead of
updating member variable
@VisibleForTesting
synchronized void populateCounterUpdates(WorkItemStatus status) {
if (worker == null) {
return;
}
+// Checking against boolean, because getCompleted can return null?
boolean isFinalUpdate = Boolean.TRUE.equals(status.getCompleted());
-ImmutableList.Builder counterUpdatesListBuilder =
ImmutableList.builder();
+Map counterUpdatesMap = new HashMap<>();
// Output counters
-
counterUpdatesListBuilder.addAll(extractCounters(worker.getOutputCounters()));
+extractCounters(worker.getOutputCounters())
+.forEach(
+x ->
+counterUpdatesMap.put(
+x.getStructuredNameAndMetadata() == null
+? x.getNameAndKind()
+: x.getStructuredNameAndMetadata(),
+x));
+
// User metrics reported in Worker
-counterUpdatesListBuilder.addAll(extractMetrics(isFinalUpdate));
+extractMetrics(isFinalUpdate)
+.forEach(
+x ->
+counterUpdatesMap.put(
+x.getStructuredNameAndMetadata() == null
+? x.getNameAndKind()
+: x.getStructuredNameAndMetadata(),
+x));
+
+// counterUpdatesListBuilder.addAll(extractMetrics(isFinalUpdate));
// MSec counters reported in worker
-counterUpdatesListBuilder.addAll(extractMsecCounters(isFinalUpdate));
-// Metrics reported in SDK runner.
-counterUpdatesListBuilder.addAll(worker.extractMetricUpdates());
+extractMsecCounters(isFinalUpdate)
+.forEach(
+x ->
+counterUpdatesMap.put(
+x.getStructuredNameAndMetadata() == null
+? x.getNameAndKind()
+: x.getStructuredNameAndMetadata(),
+x));
-ImmutableList counterUpdates =
counterUpdatesListBuilder.build();
Review comment:
This is not an error in this implementation.
I overwrite all metrics that are defined within java worker with metrics
from SDK intentionally.
Do you mean to log warning every time we overwrite CounterUpdate in case of
coding error?
It should be transparent regardless of mode we are running in.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 178740)
Time Spent: 10h 40m (was: 10.5h)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 10h 40m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=178745&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-178745
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 26/Dec/18 19:33
Start Date: 26/Dec/18 19:33
Worklog Time Spent: 10m
Work Description: Ardagan commented on pull request #7323: [BEAM-6181]
Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r244039156
##
File path:
runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/SpecMonitoringInfoValidatorTest.java
##
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.metrics;
+
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo;
+import org.junit.Before;
+import org.junit.Test;
+
+/** Relevant tests. */
+public class SpecMonitoringInfoValidatorTest {
+
+ SpecMonitoringInfoValidator testObject = null;
+
+ @Before
+ public void setUp() throws Exception {
+testObject = new SpecMonitoringInfoValidator();
Review comment:
It is one of approaches to implementing unittests, where you name object
you're testing explicitly as testObject. This usually simplifies reading test
since you always know what operations are called for tested obect and which are
part of setup.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 178745)
Time Spent: 11h 10m (was: 11h)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 11h 10m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=178741&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-178741
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 26/Dec/18 19:05
Start Date: 26/Dec/18 19:05
Worklog Time Spent: 10m
Work Description: Ardagan commented on pull request #7323: [BEAM-6181]
Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r244036064
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/UserMonitoringInfoToCounterUpdateTransformer.java
##
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.fn.control;
+
+import com.google.api.services.dataflow.model.CounterMetadata;
+import com.google.api.services.dataflow.model.CounterStructuredName;
+import com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata;
+import com.google.api.services.dataflow.model.CounterUpdate;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfoSpecs.Enum;
+import org.apache.beam.runners.core.metrics.SpecMonitoringInfoValidator;
+import
org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowStepContext;
+import
org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Origin;
+import
org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Class for transforming MonitoringInfo's containing User counter values, to
relevant CounterUpdate
+ * proto.
+ */
+class UserMonitoringInfoToCounterUpdateTransformer
+implements MonitoringInfoToCounterUpdateTransformer {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(BeamFnMapTaskExecutor.class);
+
+ private final Map transformIdMapping;
+
+ private final SpecMonitoringInfoValidator specValidator;
+
+ public UserMonitoringInfoToCounterUpdateTransformer(
+ final SpecMonitoringInfoValidator specMonitoringInfoValidator,
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+this.specValidator = specMonitoringInfoValidator;
+ }
+
+ static final String BEAM_METRICS_USER_PREFIX =
Review comment:
I see it as this value is defined in one place -- beam_fn_api.proto and
both, SimpleMonitoringInfoBuilder and
UserMonitoringInfoToCounterUpdateTransformer only define convenient accessor to
the field.
I think that creating a structure with these fields separate from proto will
only bring more inconvenience to the code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 178741)
Time Spent: 10h 50m (was: 10h 40m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 10h 50m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=178743&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-178743
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 26/Dec/18 19:13
Start Date: 26/Dec/18 19:13
Worklog Time Spent: 10m
Work Description: Ardagan commented on pull request #7323: [BEAM-6181]
Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r244037013
##
File path:
runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/SpecMonitoringInfoValidatorTest.java
##
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.metrics;
+
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo;
+import org.junit.Before;
+import org.junit.Test;
+
+/** Relevant tests. */
+public class SpecMonitoringInfoValidatorTest {
+
+ SpecMonitoringInfoValidator testObject = null;
+
+ @Before
+ public void setUp() throws Exception {
+testObject = new SpecMonitoringInfoValidator();
+ }
+
+ @Test
+ public void validateReturnsErrorOnInvalidMonitoringInfoType() {
+MonitoringInfo testInput =
+MonitoringInfo.newBuilder()
+.setUrn("beam:metric:user:someCounter")
+.setType("beam:metrics:bad_value")
+.build();
+assertTrue(testObject.validate(testInput).isPresent());
+ }
+
+ @Test
+ public void validateReturnsNoErrorOnValidMonitoringInfo() {
+MonitoringInfo testInput =
+MonitoringInfo.newBuilder()
+.setUrn("beam:metric:user:someCounter")
+.setType("beam:metrics:sum_int_64")
+.putLabels("dummy", "value")
+.build();
+assertFalse(testObject.validate(testInput).isPresent());
+
+testInput =
+MonitoringInfo.newBuilder()
+.setUrn("beam:metric:element_count:v1")
Review comment:
I was thinking of this, but since these constants will be declared outside
of test body, it will make test less readable.
Do you think it is worth doing?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 178743)
Time Spent: 11h (was: 10h 50m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 11h
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=178736&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-178736
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 26/Dec/18 18:30
Start Date: 26/Dec/18 18:30
Worklog Time Spent: 10m
Work Description: Ardagan commented on pull request #7323: [BEAM-6181]
Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r244032255
##
File path:
runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/SimpleMonitoringInfoBuilder.java
##
@@ -163,8 +111,9 @@ private static String userMetricUrn(String
metricNamespace, String metricName) {
*
* @param urn The urn of the MonitoringInfo
*/
- public void setUrn(String urn) {
+ public SimpleMonitoringInfoBuilder setUrn(String urn) {
Review comment:
Thank you.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 178736)
Time Spent: 10.5h (was: 10h 20m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 10.5h
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=178735&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-178735
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 26/Dec/18 18:29
Start Date: 26/Dec/18 18:29
Worklog Time Spent: 10m
Work Description: Ardagan commented on pull request #7323: [BEAM-6181]
Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r244032235
##
File path:
runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/SpecMonitoringInfoValidatorTest.java
##
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.metrics;
+
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo;
+import org.junit.Before;
+import org.junit.Test;
+
+/** Relevant tests. */
+public class SpecMonitoringInfoValidatorTest {
+
+ SpecMonitoringInfoValidator testObject = null;
+
+ @Before
+ public void setUp() throws Exception {
+testObject = new SpecMonitoringInfoValidator();
+ }
+
+ @Test
+ public void validateReturnsErrorOnInvalidMonitoringInfoType() {
Review comment:
SpecMonitoringInfoValidator has only one method "validate", so I skip it in
test names.
Test verifies that "validate" method returns error on invalid MonitoringInfo
type received.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 178735)
Time Spent: 10h 20m (was: 10h 10m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 10h 20m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177786&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177786
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 20/Dec/18 22:22
Start Date: 20/Dec/18 22:22
Worklog Time Spent: 10m
Work Description: Ardagan commented on a change in pull request #7323:
[BEAM-6181] Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r243436359
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/MSecMonitoringInfoToCounterUpdateTransformer.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.fn.control;
+
+import com.google.api.services.dataflow.model.CounterMetadata;
+import com.google.api.services.dataflow.model.CounterStructuredName;
+import com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata;
+import com.google.api.services.dataflow.model.CounterUpdate;
+import com.google.common.annotations.VisibleForTesting;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo;
+import org.apache.beam.runners.core.metrics.SpecMonitoringInfoValidator;
+import
org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowStepContext;
+import
org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class MSecMonitoringInfoToCounterUpdateTransformer
+implements MonitoringInfoToCounterUpdateTransformer {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(BeamFnMapTaskExecutor.class);
+
+ private SpecMonitoringInfoValidator specValidator;
+ private Map transformIdMapping;
+ private Map urnToCounterNameMapping;
+
+ /**
+ * @param specValidator SpecMonitoringInfoValidator to utilize for default
validation.
+ * @param transformIdMapping Mapping of PTransform ID string to
DataflowStepContext.
+ */
+ public MSecMonitoringInfoToCounterUpdateTransformer(
+ SpecMonitoringInfoValidator specValidator,
+ Map transformIdMapping) {
+this.specValidator = specValidator;
+this.transformIdMapping = transformIdMapping;
+urnToCounterNameMapping = createKnownUrnToCounterNameMapping();
+ }
+
+ /** Allows to inject members for cleaner testing. */
+ @VisibleForTesting
+ protected MSecMonitoringInfoToCounterUpdateTransformer(
+ SpecMonitoringInfoValidator specValidator,
+ Map transformIdMapping,
+ Map urnToCounterNameMapping) {
+this.specValidator = specValidator;
+this.transformIdMapping = transformIdMapping;
+this.urnToCounterNameMapping = urnToCounterNameMapping;
+ }
+
+ @VisibleForTesting
+ protected Map createKnownUrnToCounterNameMapping() {
+Map result = new HashMap<>();
+result.put("beam:metric:pardo_execution_time:start_bundle_msecs:v1",
"start-msecs");
+result.put("beam:metric:pardo_execution_time:process_bundle_msecs:v1",
"process-msecs");
+result.put("beam:metric:pardo_execution_time:finish_bundle_msecs:v1",
"finish-msecs");
+return result;
+ }
+
+ /**
+ * Validates provided monitoring info against specs and common safety checks.
+ *
+ * @param monitoringInfo to validate.
+ * @return Optional.empty() all validation checks are passed. Optional with
error text otherwise.
+ * @throws RuntimeException if received unexpected urn.
+ */
+ protected Optional validate(MonitoringInfo monitoringInfo) {
+Optional validatorResult = specValidator.validate(monitoringInfo);
+if (validatorResult.isPresent()) {
+ return validatorResult;
+}
+
+String urn = monitoringInfo.getUrn();
+if (!urnToCounterNameMapping.keySet().contains(urn)) {
+ throw new RuntimeException(String.format("Received unexpected counter
urn: %s", urn));
Review comment:
This exception doesn't validate MonitoringInfo, instead it che
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177735&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177735
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 20/Dec/18 19:45
Start Date: 20/Dec/18 19:45
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7323:
[BEAM-6181] Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r243392154
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/MSecMonitoringInfoToCounterUpdateTransformer.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.fn.control;
+
+import com.google.api.services.dataflow.model.CounterMetadata;
+import com.google.api.services.dataflow.model.CounterStructuredName;
+import com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata;
+import com.google.api.services.dataflow.model.CounterUpdate;
+import com.google.common.annotations.VisibleForTesting;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo;
+import org.apache.beam.runners.core.metrics.SpecMonitoringInfoValidator;
+import
org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowStepContext;
+import
org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class MSecMonitoringInfoToCounterUpdateTransformer
+implements MonitoringInfoToCounterUpdateTransformer {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(BeamFnMapTaskExecutor.class);
+
+ private SpecMonitoringInfoValidator specValidator;
+ private Map transformIdMapping;
+ private Map urnToCounterNameMapping;
+
+ /**
+ * @param specValidator SpecMonitoringInfoValidator to utilize for default
validation.
+ * @param transformIdMapping Mapping of PTransform ID string to
DataflowStepContext.
+ */
+ public MSecMonitoringInfoToCounterUpdateTransformer(
+ SpecMonitoringInfoValidator specValidator,
+ Map transformIdMapping) {
+this.specValidator = specValidator;
+this.transformIdMapping = transformIdMapping;
+urnToCounterNameMapping = createKnownUrnToCounterNameMapping();
+ }
+
+ /** Allows to inject members for cleaner testing. */
+ @VisibleForTesting
+ protected MSecMonitoringInfoToCounterUpdateTransformer(
+ SpecMonitoringInfoValidator specValidator,
+ Map transformIdMapping,
+ Map urnToCounterNameMapping) {
+this.specValidator = specValidator;
+this.transformIdMapping = transformIdMapping;
+this.urnToCounterNameMapping = urnToCounterNameMapping;
+ }
+
+ @VisibleForTesting
+ protected Map createKnownUrnToCounterNameMapping() {
+Map result = new HashMap<>();
+result.put("beam:metric:pardo_execution_time:start_bundle_msecs:v1",
"start-msecs");
+result.put("beam:metric:pardo_execution_time:process_bundle_msecs:v1",
"process-msecs");
+result.put("beam:metric:pardo_execution_time:finish_bundle_msecs:v1",
"finish-msecs");
+return result;
+ }
+
+ /**
+ * Validates provided monitoring info against specs and common safety checks.
+ *
+ * @param monitoringInfo to validate.
+ * @return Optional.empty() all validation checks are passed. Optional with
error text otherwise.
+ * @throws RuntimeException if received unexpected urn.
+ */
+ protected Optional validate(MonitoringInfo monitoringInfo) {
+Optional validatorResult = specValidator.validate(monitoringInfo);
+if (validatorResult.isPresent()) {
+ return validatorResult;
+}
+
+String urn = monitoringInfo.getUrn();
+if (!urnToCounterNameMapping.keySet().contains(urn)) {
+ throw new RuntimeException(String.format("Received unexpected counter
urn: %s", urn));
+}
+
+final String ptransform = monitoringInfo.getLabelsMap().get("PTRANSFOR
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177727&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177727
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 20/Dec/18 19:45
Start Date: 20/Dec/18 19:45
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7323:
[BEAM-6181] Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r243385704
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/UserMonitoringInfoToCounterUpdateTransformer.java
##
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.fn.control;
+
+import com.google.api.services.dataflow.model.CounterMetadata;
+import com.google.api.services.dataflow.model.CounterStructuredName;
+import com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata;
+import com.google.api.services.dataflow.model.CounterUpdate;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfoSpecs.Enum;
+import org.apache.beam.runners.core.metrics.SpecMonitoringInfoValidator;
+import
org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowStepContext;
+import
org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Origin;
+import
org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Class for transforming MonitoringInfo's containing User counter values, to
relevant CounterUpdate
+ * proto.
+ */
+class UserMonitoringInfoToCounterUpdateTransformer
+implements MonitoringInfoToCounterUpdateTransformer {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(BeamFnMapTaskExecutor.class);
+
+ private final Map transformIdMapping;
+
+ private final SpecMonitoringInfoValidator specValidator;
+
+ public UserMonitoringInfoToCounterUpdateTransformer(
+ final SpecMonitoringInfoValidator specMonitoringInfoValidator,
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+this.specValidator = specMonitoringInfoValidator;
+ }
+
+ static final String BEAM_METRICS_USER_PREFIX =
Review comment:
This is defined in two files now, SimpleMonitoringInfoBuilder. Can we just
define it in one place
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 177727)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 9h 40m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177724&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177724
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 20/Dec/18 19:45
Start Date: 20/Dec/18 19:45
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7323:
[BEAM-6181] Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r243394709
##
File path:
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/MSecMonitoringInfoToCounterUpdateTransformerTest.java
##
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.fn.control;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.mockito.Matchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import com.google.api.services.dataflow.model.CounterUpdate;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo;
+import org.apache.beam.runners.core.metrics.SpecMonitoringInfoValidator;
+import
org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowStepContext;
+import org.apache.beam.runners.dataflow.worker.counters.NameContext;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.mockito.Mock;
+import org.mockito.MockitoAnnotations;
+
+public class MSecMonitoringInfoToCounterUpdateTransformerTest {
+
+ @Rule public final ExpectedException exception = ExpectedException.none();
+
+ @Mock private SpecMonitoringInfoValidator mockSpecValidator;
+
+ @Before
+ public void setUp() throws Exception {
+MockitoAnnotations.initMocks(this);
+ }
+
+ @Test
+ public void testTransformReturnsNullIfSpecValidationFails() {
+Map counterNameMapping = new HashMap<>();
+counterNameMapping.put("beam:counter:supported", "supportedCounter");
+
+Map stepContextMapping = new HashMap<>();
+
+MSecMonitoringInfoToCounterUpdateTransformer testObject =
+new MSecMonitoringInfoToCounterUpdateTransformer(
+mockSpecValidator, stepContextMapping, counterNameMapping);
+
+Optional error = Optional.of("Error text");
+when(mockSpecValidator.validate(any())).thenReturn(error);
+
+MonitoringInfo monitoringInfo =
+MonitoringInfo.newBuilder()
+
.setUrn("beam:metric:pardo_execution_time:start_bundle_msecs:v1:invalid")
+.build();
+assertEquals(null, testObject.transform(monitoringInfo));
+ }
+
+ @Test
+ public void testTransformThrowsIfMonitoringInfoWithUnknownUrnReceived() {
+Map counterNameMapping = new HashMap<>();
+counterNameMapping.put("beam:counter:supported", "supportedCounter");
+
+Map stepContextMapping = new HashMap<>();
+MonitoringInfo monitoringInfo =
+MonitoringInfo.newBuilder()
+
.setUrn("beam:metric:pardo_execution_time:start_bundle_msecs:v1:invalid")
+.build();
+
+MSecMonitoringInfoToCounterUpdateTransformer testObject =
+new MSecMonitoringInfoToCounterUpdateTransformer(
+mockSpecValidator, stepContextMapping, counterNameMapping);
+
+when(mockSpecValidator.validate(any())).thenReturn(Optional.empty());
+
+exception.expect(RuntimeException.class);
+testObject.transform(monitoringInfo);
+ }
+
+ @Test
+ public void
testTransformThrowsIfMonitoringInfoWithUnknownPTransformLabelPresent() {
+Map counterNameMapping = new HashMap<>();
+counterNameMapping.put("beam:counter:supported", "supportedCounter");
+
+Map stepContextMapping = new HashMap<>();
+
+MSecMonitoringInfoToCounterUpdateTransformer testObject =
+new MSecMonitoringInfoToCounterUpdateTransformer(mockSpecValidator,
stepContextMapping);
+
+when(mockSpecValidator.validate(any())).the
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177736&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177736
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 20/Dec/18 19:45
Start Date: 20/Dec/18 19:45
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7323:
[BEAM-6181] Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r243373285
##
File path:
runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/SpecMonitoringInfoValidatorTest.java
##
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.metrics;
+
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo;
+import org.junit.Before;
+import org.junit.Test;
+
+/** Relevant tests. */
+public class SpecMonitoringInfoValidatorTest {
+
+ SpecMonitoringInfoValidator testObject = null;
+
+ @Before
+ public void setUp() throws Exception {
+testObject = new SpecMonitoringInfoValidator();
+ }
+
+ @Test
+ public void validateReturnsErrorOnInvalidMonitoringInfoType() {
+MonitoringInfo testInput =
+MonitoringInfo.newBuilder()
+.setUrn("beam:metric:user:someCounter")
+.setType("beam:metrics:bad_value")
+.build();
+assertTrue(testObject.validate(testInput).isPresent());
+ }
+
+ @Test
+ public void validateReturnsNoErrorOnValidMonitoringInfo() {
+MonitoringInfo testInput =
+MonitoringInfo.newBuilder()
+.setUrn("beam:metric:user:someCounter")
+.setType("beam:metrics:sum_int_64")
+.putLabels("dummy", "value")
+.build();
+assertFalse(testObject.validate(testInput).isPresent());
+
+testInput =
+MonitoringInfo.newBuilder()
+.setUrn("beam:metric:element_count:v1")
Review comment:
Can you use the constants please
SimpleMonitoringInfoBuilder.ELEMENT_COUNT_URN
SimpleMonitoringInfoBuilder.USER_COUNTER_PREFIX
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 177736)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 10h
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177737&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177737
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 20/Dec/18 19:45
Start Date: 20/Dec/18 19:45
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7323:
[BEAM-6181] Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r243388929
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/MSecMonitoringInfoToCounterUpdateTransformer.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.fn.control;
+
+import com.google.api.services.dataflow.model.CounterMetadata;
+import com.google.api.services.dataflow.model.CounterStructuredName;
+import com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata;
+import com.google.api.services.dataflow.model.CounterUpdate;
+import com.google.common.annotations.VisibleForTesting;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo;
+import org.apache.beam.runners.core.metrics.SpecMonitoringInfoValidator;
+import
org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowStepContext;
+import
org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class MSecMonitoringInfoToCounterUpdateTransformer
+implements MonitoringInfoToCounterUpdateTransformer {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(BeamFnMapTaskExecutor.class);
+
+ private SpecMonitoringInfoValidator specValidator;
+ private Map transformIdMapping;
+ private Map urnToCounterNameMapping;
+
+ /**
+ * @param specValidator SpecMonitoringInfoValidator to utilize for default
validation.
+ * @param transformIdMapping Mapping of PTransform ID string to
DataflowStepContext.
+ */
+ public MSecMonitoringInfoToCounterUpdateTransformer(
+ SpecMonitoringInfoValidator specValidator,
+ Map transformIdMapping) {
+this.specValidator = specValidator;
+this.transformIdMapping = transformIdMapping;
+urnToCounterNameMapping = createKnownUrnToCounterNameMapping();
+ }
+
+ /** Allows to inject members for cleaner testing. */
+ @VisibleForTesting
+ protected MSecMonitoringInfoToCounterUpdateTransformer(
+ SpecMonitoringInfoValidator specValidator,
+ Map transformIdMapping,
+ Map urnToCounterNameMapping) {
+this.specValidator = specValidator;
+this.transformIdMapping = transformIdMapping;
+this.urnToCounterNameMapping = urnToCounterNameMapping;
+ }
+
+ @VisibleForTesting
+ protected Map createKnownUrnToCounterNameMapping() {
+Map result = new HashMap<>();
+result.put("beam:metric:pardo_execution_time:start_bundle_msecs:v1",
"start-msecs");
+result.put("beam:metric:pardo_execution_time:process_bundle_msecs:v1",
"process-msecs");
+result.put("beam:metric:pardo_execution_time:finish_bundle_msecs:v1",
"finish-msecs");
+return result;
+ }
+
+ /**
+ * Validates provided monitoring info against specs and common safety checks.
+ *
+ * @param monitoringInfo to validate.
+ * @return Optional.empty() all validation checks are passed. Optional with
error text otherwise.
+ * @throws RuntimeException if received unexpected urn.
+ */
+ protected Optional validate(MonitoringInfo monitoringInfo) {
+Optional validatorResult = specValidator.validate(monitoringInfo);
+if (validatorResult.isPresent()) {
+ return validatorResult;
+}
+
+String urn = monitoringInfo.getUrn();
+if (!urnToCounterNameMapping.keySet().contains(urn)) {
+ throw new RuntimeException(String.format("Received unexpected counter
urn: %s", urn));
Review comment:
Please just drop metrics if they are invalid, don't throw an e
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177732&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177732
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 20/Dec/18 19:45
Start Date: 20/Dec/18 19:45
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7323:
[BEAM-6181] Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r243374184
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkItemStatusClient.java
##
@@ -266,27 +266,62 @@ private synchronized WorkItemStatus
createStatusUpdate(boolean isFinal) {
return status;
}
- // todo(migryz) this method should return List instead of
updating member variable
@VisibleForTesting
synchronized void populateCounterUpdates(WorkItemStatus status) {
if (worker == null) {
return;
}
+// Checking against boolean, because getCompleted can return null?
boolean isFinalUpdate = Boolean.TRUE.equals(status.getCompleted());
-ImmutableList.Builder counterUpdatesListBuilder =
ImmutableList.builder();
+Map counterUpdatesMap = new HashMap<>();
// Output counters
-
counterUpdatesListBuilder.addAll(extractCounters(worker.getOutputCounters()));
+extractCounters(worker.getOutputCounters())
+.forEach(
+x ->
+counterUpdatesMap.put(
+x.getStructuredNameAndMetadata() == null
+? x.getNameAndKind()
+: x.getStructuredNameAndMetadata(),
+x));
+
// User metrics reported in Worker
-counterUpdatesListBuilder.addAll(extractMetrics(isFinalUpdate));
+extractMetrics(isFinalUpdate)
+.forEach(
+x ->
+counterUpdatesMap.put(
+x.getStructuredNameAndMetadata() == null
+? x.getNameAndKind()
+: x.getStructuredNameAndMetadata(),
+x));
+
+// counterUpdatesListBuilder.addAll(extractMetrics(isFinalUpdate));
// MSec counters reported in worker
-counterUpdatesListBuilder.addAll(extractMsecCounters(isFinalUpdate));
-// Metrics reported in SDK runner.
-counterUpdatesListBuilder.addAll(worker.extractMetricUpdates());
+extractMsecCounters(isFinalUpdate)
+.forEach(
+x ->
+counterUpdatesMap.put(
+x.getStructuredNameAndMetadata() == null
+? x.getNameAndKind()
+: x.getStructuredNameAndMetadata(),
+x));
-ImmutableList counterUpdates =
counterUpdatesListBuilder.build();
Review comment:
Can you log error if we try to add a duplicate to this map. This is one of
the failure cases you were afraid of. It would be better if we could add a
flag/experiment or something around the new functionality and only enable it in
the portable runner. I don't want this to accidentally break the existing code.
Is there a way to know this code is running portability mode? If you know
that then you can accept metrics conditionaly from diff sources
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 177732)
Time Spent: 9h 50m (was: 9h 40m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 9h 50m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177726&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177726
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 20/Dec/18 19:45
Start Date: 20/Dec/18 19:45
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7323:
[BEAM-6181] Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r243372441
##
File path:
runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/SimpleMonitoringInfoBuilder.java
##
@@ -163,8 +111,9 @@ private static String userMetricUrn(String
metricNamespace, String metricName) {
*
* @param urn The urn of the MonitoringInfo
*/
- public void setUrn(String urn) {
+ public SimpleMonitoringInfoBuilder setUrn(String urn) {
Review comment:
Nice improvement to follow proper builder pattern, building has been
simplified a lot.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 177726)
Time Spent: 9h 40m (was: 9.5h)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 9h 40m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177729&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177729
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 20/Dec/18 19:45
Start Date: 20/Dec/18 19:45
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7323:
[BEAM-6181] Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r243372746
##
File path:
runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/SpecMonitoringInfoValidatorTest.java
##
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.metrics;
+
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo;
+import org.junit.Before;
+import org.junit.Test;
+
+/** Relevant tests. */
+public class SpecMonitoringInfoValidatorTest {
+
+ SpecMonitoringInfoValidator testObject = null;
+
+ @Before
+ public void setUp() throws Exception {
+testObject = new SpecMonitoringInfoValidator();
+ }
+
+ @Test
+ public void validateReturnsErrorOnInvalidMonitoringInfoType() {
Review comment:
Just seems like the tests are a bit inverted.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 177729)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 9h 40m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177733&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177733
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 20/Dec/18 19:45
Start Date: 20/Dec/18 19:45
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7323:
[BEAM-6181] Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r243371541
##
File path:
runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/SpecMonitoringInfoValidator.java
##
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.metrics;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfoSpec;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfoSpecs;
+
+/** Class implements validation of MonitoringInfos against
MonitoringInfoSpecs. */
+public class SpecMonitoringInfoValidator {
+ protected final MonitoringInfoSpec[] specs;
+
+ public SpecMonitoringInfoValidator() {
+specs =
+Arrays.stream(MonitoringInfoSpecs.Enum.values())
+.filter(x -> !(x).name().equals("UNRECOGNIZED"))
Review comment:
Please add comment back describing why filter out using this magic string
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 177733)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 9h 50m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177725&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177725
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 20/Dec/18 19:45
Start Date: 20/Dec/18 19:45
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7323:
[BEAM-6181] Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r243372104
##
File path:
runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/SpecMonitoringInfoValidatorTest.java
##
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.metrics;
+
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo;
+import org.junit.Before;
+import org.junit.Test;
+
+/** Relevant tests. */
+public class SpecMonitoringInfoValidatorTest {
+
+ SpecMonitoringInfoValidator testObject = null;
+
+ @Before
+ public void setUp() throws Exception {
+testObject = new SpecMonitoringInfoValidator();
+ }
+
+ @Test
+ public void validateReturnsErrorOnInvalidMonitoringInfoType() {
Review comment:
Does this test return an error? Seems like it is successful
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 177725)
Time Spent: 9h 40m (was: 9.5h)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 9h 40m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177730&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177730
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 20/Dec/18 19:45
Start Date: 20/Dec/18 19:45
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7323:
[BEAM-6181] Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r243374422
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -356,6 +352,7 @@ void updateProgress() {
updateMetricsDeprecated(metrics);
// todo(migryz): utilize monitoringInfos here.
+// Requires Count metrics to be implemented.
Review comment:
Element Count
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 177730)
Time Spent: 9h 50m (was: 9h 40m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 9h 50m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177734&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177734
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 20/Dec/18 19:45
Start Date: 20/Dec/18 19:45
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7323:
[BEAM-6181] Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r243371969
##
File path:
runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/SpecMonitoringInfoValidatorTest.java
##
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.metrics;
+
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo;
+import org.junit.Before;
+import org.junit.Test;
+
+/** Relevant tests. */
+public class SpecMonitoringInfoValidatorTest {
+
+ SpecMonitoringInfoValidator testObject = null;
+
+ @Before
+ public void setUp() throws Exception {
+testObject = new SpecMonitoringInfoValidator();
Review comment:
rename test object to validator please
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 177734)
Time Spent: 10h (was: 9h 50m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 10h
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177728&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177728
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 20/Dec/18 19:45
Start Date: 20/Dec/18 19:45
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7323:
[BEAM-6181] Implemented msec counters support in FnApi world.
URL: https://github.com/apache/beam/pull/7323#discussion_r243373680
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkItemStatusClient.java
##
@@ -266,27 +266,62 @@ private synchronized WorkItemStatus
createStatusUpdate(boolean isFinal) {
return status;
}
- // todo(migryz) this method should return List instead of
updating member variable
@VisibleForTesting
synchronized void populateCounterUpdates(WorkItemStatus status) {
if (worker == null) {
return;
}
+// Checking against boolean, because getCompleted can return null?
boolean isFinalUpdate = Boolean.TRUE.equals(status.getCompleted());
-ImmutableList.Builder counterUpdatesListBuilder =
ImmutableList.builder();
+Map counterUpdatesMap = new HashMap<>();
// Output counters
-
counterUpdatesListBuilder.addAll(extractCounters(worker.getOutputCounters()));
+extractCounters(worker.getOutputCounters())
+.forEach(
+x ->
+counterUpdatesMap.put(
+x.getStructuredNameAndMetadata() == null
+? x.getNameAndKind()
+: x.getStructuredNameAndMetadata(),
+x));
+
// User metrics reported in Worker
-counterUpdatesListBuilder.addAll(extractMetrics(isFinalUpdate));
+extractMetrics(isFinalUpdate)
+.forEach(
+x ->
+counterUpdatesMap.put(
+x.getStructuredNameAndMetadata() == null
+? x.getNameAndKind()
+: x.getStructuredNameAndMetadata(),
+x));
+
+// counterUpdatesListBuilder.addAll(extractMetrics(isFinalUpdate));
// MSec counters reported in worker
-counterUpdatesListBuilder.addAll(extractMsecCounters(isFinalUpdate));
-// Metrics reported in SDK runner.
-counterUpdatesListBuilder.addAll(worker.extractMetricUpdates());
+extractMsecCounters(isFinalUpdate)
+.forEach(
+x ->
+counterUpdatesMap.put(
Review comment:
This code is repeated a lot, can you save this lambda and reuse it? Seems
identical, isn't it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 177728)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 9h 40m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177731&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177731 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 20/Dec/18 19:45 Start Date: 20/Dec/18 19:45 Worklog Time Spent: 10m Work Description: ajamato commented on a change in pull request #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#discussion_r243390757 ## File path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/MSecMonitoringInfoToCounterUpdateTransformer.java ## @@ -0,0 +1,139 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.beam.runners.dataflow.worker.fn.control; + +import com.google.api.services.dataflow.model.CounterMetadata; +import com.google.api.services.dataflow.model.CounterStructuredName; +import com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata; +import com.google.api.services.dataflow.model.CounterUpdate; +import com.google.common.annotations.VisibleForTesting; +import java.util.HashMap; +import java.util.Map; +import java.util.Optional; +import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo; +import org.apache.beam.runners.core.metrics.SpecMonitoringInfoValidator; +import org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowStepContext; +import org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class MSecMonitoringInfoToCounterUpdateTransformer Review comment: Please add class doc comment This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 177731) Time Spent: 9h 50m (was: 9h 40m) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 9h 50m > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177631&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177631 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 20/Dec/18 17:36 Start Date: 20/Dec/18 17:36 Worklog Time Spent: 10m Work Description: Ardagan commented on issue #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#issuecomment-449077415 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 177631) Time Spent: 9h 20m (was: 9h 10m) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 9h 20m > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177629&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177629 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 20/Dec/18 17:27 Start Date: 20/Dec/18 17:27 Worklog Time Spent: 10m Work Description: Ardagan commented on issue #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#issuecomment-449074735 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 177629) Time Spent: 9h 10m (was: 9h) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 9h 10m > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177628&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177628 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 20/Dec/18 17:25 Start Date: 20/Dec/18 17:25 Worklog Time Spent: 10m Work Description: Ardagan removed a comment on issue #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#issuecomment-449073946 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 177628) Time Spent: 9h (was: 8h 50m) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 9h > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177626&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177626 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 20/Dec/18 17:24 Start Date: 20/Dec/18 17:24 Worklog Time Spent: 10m Work Description: Ardagan commented on issue #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#issuecomment-449073946 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 177626) Time Spent: 8h 50m (was: 8h 40m) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 8h 50m > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177616&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177616 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 20/Dec/18 17:18 Start Date: 20/Dec/18 17:18 Worklog Time Spent: 10m Work Description: Ardagan removed a comment on issue #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#issuecomment-449071533 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 177616) Time Spent: 8h 40m (was: 8.5h) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 8h 40m > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177615&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177615 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 20/Dec/18 17:17 Start Date: 20/Dec/18 17:17 Worklog Time Spent: 10m Work Description: Ardagan commented on issue #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#issuecomment-449071533 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 177615) Time Spent: 8.5h (was: 8h 20m) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 8.5h > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177593&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177593 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 20/Dec/18 17:01 Start Date: 20/Dec/18 17:01 Worklog Time Spent: 10m Work Description: Ardagan commented on issue #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#issuecomment-449066679 run Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 177593) Time Spent: 8h (was: 7h 50m) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 8h > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177595&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177595 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 20/Dec/18 17:01 Start Date: 20/Dec/18 17:01 Worklog Time Spent: 10m Work Description: Ardagan removed a comment on issue #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#issuecomment-449066679 run Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 177595) Time Spent: 8h 20m (was: 8h 10m) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 8h 20m > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177594&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177594 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 20/Dec/18 17:01 Start Date: 20/Dec/18 17:01 Worklog Time Spent: 10m Work Description: Ardagan removed a comment on issue #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#issuecomment-449065854 run Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 177594) Time Spent: 8h 10m (was: 8h) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 8h 10m > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177589&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177589 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 20/Dec/18 16:58 Start Date: 20/Dec/18 16:58 Worklog Time Spent: 10m Work Description: Ardagan commented on issue #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#issuecomment-449065854 run Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 177589) Time Spent: 7h 50m (was: 7h 40m) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 7h 50m > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177140&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177140 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 19/Dec/18 19:06 Start Date: 19/Dec/18 19:06 Worklog Time Spent: 10m Work Description: Ardagan commented on issue #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323#issuecomment-448709078 @ajamato This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 177140) Time Spent: 7h 40m (was: 7.5h) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 7h 40m > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=177135&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177135 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 19/Dec/18 18:58 Start Date: 19/Dec/18 18:58 Worklog Time Spent: 10m Work Description: Ardagan opened a new pull request #7323: [BEAM-6181] Implemented msec counters support in FnApi world. URL: https://github.com/apache/beam/pull/7323 Minor refactoring to generalize validation and counter transformation. Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 177135) Time Spent: 7.5h (was: 7h 20m) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEA
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=174275&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-174275
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 11/Dec/18 20:58
Start Date: 11/Dec/18 20:58
Worklog Time Spent: 10m
Work Description: robertwb closed pull request #7250: [BEAM-6181]
Unexpected metrics should be non-fatal.
URL: https://github.com/apache/beam/pull/7250
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
index 6c1d43f952f9..1127a17242a5 100644
---
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
+++
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
@@ -424,24 +424,27 @@ private CounterUpdate
monitoringInfoToCounterUpdate(MonitoringInfo monitoringInf
// Requires https://github.com/apache/beam/pull/6799 to be merged.
if (urn.startsWith(BEAM_METRICS_USER_PREFIX)) {
if (!type.equals("beam:metrics:sum_int_64")) {
-throw new RuntimeException(
+LOG.info(
"Encountered user-counter MonitoringInfo with unexpected type."
+ "Expected: beam:metrics:sum_int_64. Received: "
+ monitoringInfo.toString());
+return null;
}
final String ptransform =
monitoringInfo.getLabelsMap().get("PTRANSFORM");
if (ptransform == null) {
-throw new RuntimeException(
+LOG.info(
"Encountered user-counter MonitoringInfo with missing
ptransformId: "
+ monitoringInfo.toString());
+return null;
}
DataflowStepContext stepContext = transformIdMapping.get(ptransform);
if (stepContext == null) {
-throw new RuntimeException(
+LOG.info(
"Encountered user-counter MonitoringInfo with unknown
ptransformId: "
+ monitoringInfo.toString());
+return null;
}
CounterStructuredNameAndMetadata name = new
CounterStructuredNameAndMetadata();
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 174275)
Time Spent: 7h 20m (was: 7h 10m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=174233&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-174233 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 11/Dec/18 19:23 Start Date: 11/Dec/18 19:23 Worklog Time Spent: 10m Work Description: ajamato commented on issue #7250: [BEAM-6181] Unexpected metrics should be non-fatal. URL: https://github.com/apache/beam/pull/7250#issuecomment-446328463 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 174233) Time Spent: 7h 10m (was: 7h) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 7h 10m > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=174230&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-174230 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 11/Dec/18 19:22 Start Date: 11/Dec/18 19:22 Worklog Time Spent: 10m Work Description: ajamato commented on issue #7250: [BEAM-6181] Unexpected metrics should be non-fatal. URL: https://github.com/apache/beam/pull/7250#issuecomment-446328180 @Ardagan This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 174230) Time Spent: 7h (was: 6h 50m) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 7h > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=174081&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-174081 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 11/Dec/18 13:23 Start Date: 11/Dec/18 13:23 Worklog Time Spent: 10m Work Description: robertwb commented on issue #7250: [BEAM-6181] Unexpected metrics should be non-fatal. URL: https://github.com/apache/beam/pull/7250#issuecomment-446200542 R: @ajamato This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 174081) Time Spent: 6h 50m (was: 6h 40m) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 6h 50m > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=174079&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-174079 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 11/Dec/18 13:23 Start Date: 11/Dec/18 13:23 Worklog Time Spent: 10m Work Description: robertwb opened a new pull request #7250: [BEAM-6181] Unexpected metrics should be non-fatal. URL: https://github.com/apache/beam/pull/7250 This unbreaks dataflow streaming on Python. Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 174079) Time Spent: 6h 40m (was: 6.5h) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > >
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172891&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172891
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 07/Dec/18 00:07
Start Date: 07/Dec/18 00:07
Worklog Time Spent: 10m
Work Description: swegner closed pull request #7202: [BEAM-6181]
Reporting user counters via MonitoringInfos in Portable Dataflow Runner.
URL: https://github.com/apache/beam/pull/7202
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkItemStatusClient.java
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkItemStatusClient.java
index 2d840e3f4356..28f239784bf7 100644
---
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkItemStatusClient.java
+++
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkItemStatusClient.java
@@ -263,6 +263,7 @@ private synchronized WorkItemStatus
createStatusUpdate(boolean isFinal) {
return status;
}
+ // todo(migryz) this method should return List instead of
updating member variable
@VisibleForTesting
synchronized void populateCounterUpdates(WorkItemStatus status) {
if (worker == null) {
@@ -270,13 +271,18 @@ synchronized void populateCounterUpdates(WorkItemStatus
status) {
}
boolean isFinalUpdate = Boolean.TRUE.equals(status.getCompleted());
-ImmutableList.Builder counterUpdatesBuilder =
ImmutableList.builder();
-counterUpdatesBuilder.addAll(extractCounters(worker.getOutputCounters()));
-counterUpdatesBuilder.addAll(extractMetrics(isFinalUpdate));
-counterUpdatesBuilder.addAll(extractMsecCounters(isFinalUpdate));
-counterUpdatesBuilder.addAll(worker.extractMetricUpdates());
-ImmutableList counterUpdates =
counterUpdatesBuilder.build();
+ImmutableList.Builder counterUpdatesListBuilder =
ImmutableList.builder();
+// Output counters
+
counterUpdatesListBuilder.addAll(extractCounters(worker.getOutputCounters()));
+// User metrics reported in Worker
+counterUpdatesListBuilder.addAll(extractMetrics(isFinalUpdate));
+// MSec counters reported in worker
+counterUpdatesListBuilder.addAll(extractMsecCounters(isFinalUpdate));
+// Metrics reported in SDK runner.
+counterUpdatesListBuilder.addAll(worker.extractMetricUpdates());
+
+ImmutableList counterUpdates =
counterUpdatesListBuilder.build();
status.setCounterUpdates(counterUpdates);
}
diff --git
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
index bc0fb54d9521..6c1d43f952f9 100644
---
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
+++
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
@@ -17,20 +17,25 @@
*/
package org.apache.beam.runners.dataflow.worker.fn.control;
+import com.google.api.services.dataflow.model.CounterMetadata;
+import com.google.api.services.dataflow.model.CounterStructuredName;
+import com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata;
import com.google.api.services.dataflow.model.CounterUpdate;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
-import com.google.common.collect.FluentIterable;
import com.google.common.collect.Iterables;
import io.opencensus.common.Scope;
import io.opencensus.trace.SpanBuilder;
import io.opencensus.trace.Tracer;
import io.opencensus.trace.Tracing;
+import java.util.ArrayList;
import java.util.Arrays;
+import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.NoSuchElementException;
+import java.util.Objects;
import java.util.concurrent.CancellationException;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Executors;
@@ -39,20 +44,27 @@
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;
import java.util.function.Consumer;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
import javax.annotation.Nullable;
import javax.annotation.concurrent
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172885&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172885 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 06/Dec/18 23:34 Start Date: 06/Dec/18 23:34 Worklog Time Spent: 10m Work Description: Ardagan commented on issue #7202: [BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow Runner. URL: https://github.com/apache/beam/pull/7202#issuecomment-445069349 Thank you Scott. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 172885) Time Spent: 6h 20m (was: 6h 10m) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 6h 20m > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172884&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172884 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 06/Dec/18 23:34 Start Date: 06/Dec/18 23:34 Worklog Time Spent: 10m Work Description: swegner commented on issue #7202: [BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow Runner. URL: https://github.com/apache/beam/pull/7202#issuecomment-445069348 Pre-commits before my fixup are green:  This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 172884) Time Spent: 6h 10m (was: 6h) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 6h 10m > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172882&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172882
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 06/Dec/18 23:33
Start Date: 06/Dec/18 23:33
Worklog Time Spent: 10m
Work Description: swegner commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239653685
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -423,32 +420,28 @@ private CounterUpdate
monitoringInfoToCounterUpdate(MonitoringInfo monitoringInf
String type = monitoringInfo.getType();
-// todomigryz: run MonitoringInfo through validation process.
-// refer to https://github.com/apache/beam/pull/6799
-
+// todo(migryz): run MonitoringInfo through Proto validation process.
+// Requires https://github.com/apache/beam/pull/6799 to be merged.
if (urn.startsWith(BEAM_METRICS_USER_PREFIX)) {
if (!type.equals("beam:metrics:sum_int_64")) {
-LOG.warn(
+throw new RuntimeException(
"Ignoring user-counter MonitoringInfo with unexpected type."
Review comment:
I pushed a commit with this fixup.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172882)
Time Spent: 5h 50m (was: 5h 40m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[ https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172883&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172883 ] ASF GitHub Bot logged work on BEAM-6181: Author: ASF GitHub Bot Created on: 06/Dec/18 23:33 Start Date: 06/Dec/18 23:33 Worklog Time Spent: 10m Work Description: swegner commented on issue #7202: [BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow Runner. URL: https://github.com/apache/beam/pull/7202#issuecomment-445069206 LGTM, will squash and merge. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 172883) Time Spent: 6h (was: 5h 50m) > Utilize MetricInfo for reporting user metrics in Portable Dataflow Java > Runner. > --- > > Key: BEAM-6181 > URL: https://issues.apache.org/jira/browse/BEAM-6181 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 6h > Remaining Estimate: 0h > > New approach to report metrics in FnApi is to utilize MetricInfo structures. > This approach is implemented in Python SDK and work is ongoing in Java SDK. > This tasks includes plumbing User metrics reported via MetricInfos through > Dataflow Java Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172881&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172881
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 06/Dec/18 23:29
Start Date: 06/Dec/18 23:29
Worklog Time Spent: 10m
Work Description: swegner commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239652977
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -423,32 +420,28 @@ private CounterUpdate
monitoringInfoToCounterUpdate(MonitoringInfo monitoringInf
String type = monitoringInfo.getType();
-// todomigryz: run MonitoringInfo through validation process.
-// refer to https://github.com/apache/beam/pull/6799
-
+// todo(migryz): run MonitoringInfo through Proto validation process.
+// Requires https://github.com/apache/beam/pull/6799 to be merged.
if (urn.startsWith(BEAM_METRICS_USER_PREFIX)) {
if (!type.equals("beam:metrics:sum_int_64")) {
-LOG.warn(
+throw new RuntimeException(
"Ignoring user-counter MonitoringInfo with unexpected type."
Review comment:
These error messages don't quite make sense for throwing exceptions. We're
no longer ignoring them, we're failing execution. Suggestion:
```java
throw new RuntimeException("Encountered MonitoringInfo with unexpected type.
Expected [..]")
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172881)
Time Spent: 5h 40m (was: 5.5h)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172807&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172807
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 06/Dec/18 19:49
Start Date: 06/Dec/18 19:49
Worklog Time Spent: 10m
Work Description: swegner commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239590504
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -351,7 +398,115 @@ void updateProgress() {
}
}
-private void updateMetrics(BeamFnApi.Metrics metrics) {
+// Keeping this as static class for this iteration. Will extract to
separate file and generalize
+// when more counter types are added.
+// todomigryz: define counter transformer factory
+// that can provide respective counter transformer for different type of
counters.
+// (ie RowCountCounterTranformer, MSecCounterTransformer,
UserCounterTransformer, etc)
+private static class MonitoringInfoToCounterUpdateTransformer {
+
+ private final Map transformIdMapping;
+
+ public MonitoringInfoToCounterUpdateTransformer(
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+ }
+
+ // todo: search code for "beam:metrics"... and replace them with
relevant enums from
+ // proto after rebasing above https://github.com/apache/beam/pull/6799
that
+ // introduces relevant proto entries.
+ final String BEAM_METRICS_USER_PREFIX = "beam:metric:user";
+
+ private CounterUpdate monitoringInfoToCounterUpdate(MonitoringInfo
monitoringInfo) {
+long value =
monitoringInfo.getMetric().getCounterData().getInt64Value();
+String urn = monitoringInfo.getUrn();
+
+String type = monitoringInfo.getType();
+
+// todomigryz: run MonitoringInfo through validation process.
+// refer to https://github.com/apache/beam/pull/6799
+
+if (urn.startsWith(BEAM_METRICS_USER_PREFIX)) {
+ if (!type.equals("beam:metrics:sum_int_64")) {
+LOG.warn(
Review comment:
It would be good to understand the scenarios we could reach these states and
design error handling appropriately.
If the only scenario that would produce this is a bug in the SDK, then we
should expose that bug as early as possible to get it fixed. Log-and-continue
means that we'll ignore it, and unless we have an explicit test our users will
notice before we do.
I doubt we have a consistent methodology for error handling in Beam, but I
remember this came up when designing DIsplayData in the java SDK. We decided
errors in constructing DisplayData from user-defined PTransforms should throw
and block job submission, rather than continuing with empty Display Data. This
feels similar.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172807)
Time Spent: 5.5h (was: 5h 20m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172762&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172762
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 06/Dec/18 18:29
Start Date: 06/Dec/18 18:29
Worklog Time Spent: 10m
Work Description: Ardagan commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239541582
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -351,7 +398,115 @@ void updateProgress() {
}
}
-private void updateMetrics(BeamFnApi.Metrics metrics) {
+// Keeping this as static class for this iteration. Will extract to
separate file and generalize
+// when more counter types are added.
+// todomigryz: define counter transformer factory
+// that can provide respective counter transformer for different type of
counters.
+// (ie RowCountCounterTranformer, MSecCounterTransformer,
UserCounterTransformer, etc)
+private static class MonitoringInfoToCounterUpdateTransformer {
+
+ private final Map transformIdMapping;
+
+ public MonitoringInfoToCounterUpdateTransformer(
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+ }
+
+ // todo: search code for "beam:metrics"... and replace them with
relevant enums from
+ // proto after rebasing above https://github.com/apache/beam/pull/6799
that
+ // introduces relevant proto entries.
+ final String BEAM_METRICS_USER_PREFIX = "beam:metric:user";
+
+ private CounterUpdate monitoringInfoToCounterUpdate(MonitoringInfo
monitoringInfo) {
+long value =
monitoringInfo.getMetric().getCounterData().getInt64Value();
+String urn = monitoringInfo.getUrn();
+
+String type = monitoringInfo.getType();
+
+// todomigryz: run MonitoringInfo through validation process.
+// refer to https://github.com/apache/beam/pull/6799
+
+if (urn.startsWith(BEAM_METRICS_USER_PREFIX)) {
+ if (!type.equals("beam:metrics:sum_int_64")) {
+LOG.warn(
Review comment:
I believe that writing warning is good balance here. Most of the time you
want your job to process data, and use counters for monitoring. In this
situation usually, you do not want to lose data due to monitoring issues.
This approach allows SDK authors to find errors in their SDK implementations
on one hand. And not break user pipelines on the other hand.
@ajamato Alex, what's your opinion on this topic?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172762)
Time Spent: 5h 20m (was: 5h 10m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172714&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172714
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 06/Dec/18 17:24
Start Date: 06/Dec/18 17:24
Worklog Time Spent: 10m
Work Description: Ardagan commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239541582
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -351,7 +398,115 @@ void updateProgress() {
}
}
-private void updateMetrics(BeamFnApi.Metrics metrics) {
+// Keeping this as static class for this iteration. Will extract to
separate file and generalize
+// when more counter types are added.
+// todomigryz: define counter transformer factory
+// that can provide respective counter transformer for different type of
counters.
+// (ie RowCountCounterTranformer, MSecCounterTransformer,
UserCounterTransformer, etc)
+private static class MonitoringInfoToCounterUpdateTransformer {
+
+ private final Map transformIdMapping;
+
+ public MonitoringInfoToCounterUpdateTransformer(
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+ }
+
+ // todo: search code for "beam:metrics"... and replace them with
relevant enums from
+ // proto after rebasing above https://github.com/apache/beam/pull/6799
that
+ // introduces relevant proto entries.
+ final String BEAM_METRICS_USER_PREFIX = "beam:metric:user";
+
+ private CounterUpdate monitoringInfoToCounterUpdate(MonitoringInfo
monitoringInfo) {
+long value =
monitoringInfo.getMetric().getCounterData().getInt64Value();
+String urn = monitoringInfo.getUrn();
+
+String type = monitoringInfo.getType();
+
+// todomigryz: run MonitoringInfo through validation process.
+// refer to https://github.com/apache/beam/pull/6799
+
+if (urn.startsWith(BEAM_METRICS_USER_PREFIX)) {
+ if (!type.equals("beam:metrics:sum_int_64")) {
+LOG.warn(
Review comment:
I believe that writing warning is good balance here. Most of the time you
want your job to process data, and use counters for monitoring.
This approach allows SDK authors to find errors in their SDK implementations
on one hand. And not break user pipelines on the other hand.
@ajamato Alex, what's your opinion on this topic?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172714)
Time Spent: 5h 10m (was: 5h)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172708&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172708
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 06/Dec/18 17:20
Start Date: 06/Dec/18 17:20
Worklog Time Spent: 10m
Work Description: Ardagan commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239540159
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -351,7 +398,115 @@ void updateProgress() {
}
}
-private void updateMetrics(BeamFnApi.Metrics metrics) {
+// Keeping this as static class for this iteration. Will extract to
separate file and generalize
+// when more counter types are added.
+// todomigryz: define counter transformer factory
+// that can provide respective counter transformer for different type of
counters.
+// (ie RowCountCounterTranformer, MSecCounterTransformer,
UserCounterTransformer, etc)
+private static class MonitoringInfoToCounterUpdateTransformer {
+
+ private final Map transformIdMapping;
+
+ public MonitoringInfoToCounterUpdateTransformer(
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+ }
+
+ // todo: search code for "beam:metrics"... and replace them with
relevant enums from
Review comment:
This depends on https://github.com/apache/beam/pull/6799
Will be implemented in further PRs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172708)
Time Spent: 4h 50m (was: 4h 40m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172710&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172710
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 06/Dec/18 17:20
Start Date: 06/Dec/18 17:20
Worklog Time Spent: 10m
Work Description: Ardagan commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239538937
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -351,7 +398,115 @@ void updateProgress() {
}
}
-private void updateMetrics(BeamFnApi.Metrics metrics) {
+// Keeping this as static class for this iteration. Will extract to
separate file and generalize
+// when more counter types are added.
+// todomigryz: define counter transformer factory
+// that can provide respective counter transformer for different type of
counters.
+// (ie RowCountCounterTranformer, MSecCounterTransformer,
UserCounterTransformer, etc)
+private static class MonitoringInfoToCounterUpdateTransformer {
+
+ private final Map transformIdMapping;
+
+ public MonitoringInfoToCounterUpdateTransformer(
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+ }
+
+ // todo: search code for "beam:metrics"... and replace them with
relevant enums from
+ // proto after rebasing above https://github.com/apache/beam/pull/6799
that
+ // introduces relevant proto entries.
+ final String BEAM_METRICS_USER_PREFIX = "beam:metric:user";
+
+ private CounterUpdate monitoringInfoToCounterUpdate(MonitoringInfo
monitoringInfo) {
+long value =
monitoringInfo.getMetric().getCounterData().getInt64Value();
+String urn = monitoringInfo.getUrn();
+
+String type = monitoringInfo.getType();
+
+// todomigryz: run MonitoringInfo through validation process.
Review comment:
This requires PR mentioned on next line to go in.
(https://github.com/apache/beam/pull/6799)
Technically, I can copy data from another PR, but format of counters is
still being defined. It will cause more merging and resolving in both PRs as
well.
I suggest we keep it as is for this PR and generalize/fix it in following PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172710)
Time Spent: 5h (was: 4h 50m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 5h
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172706&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172706
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 06/Dec/18 17:16
Start Date: 06/Dec/18 17:16
Worklog Time Spent: 10m
Work Description: Ardagan commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239538937
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -351,7 +398,115 @@ void updateProgress() {
}
}
-private void updateMetrics(BeamFnApi.Metrics metrics) {
+// Keeping this as static class for this iteration. Will extract to
separate file and generalize
+// when more counter types are added.
+// todomigryz: define counter transformer factory
+// that can provide respective counter transformer for different type of
counters.
+// (ie RowCountCounterTranformer, MSecCounterTransformer,
UserCounterTransformer, etc)
+private static class MonitoringInfoToCounterUpdateTransformer {
+
+ private final Map transformIdMapping;
+
+ public MonitoringInfoToCounterUpdateTransformer(
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+ }
+
+ // todo: search code for "beam:metrics"... and replace them with
relevant enums from
+ // proto after rebasing above https://github.com/apache/beam/pull/6799
that
+ // introduces relevant proto entries.
+ final String BEAM_METRICS_USER_PREFIX = "beam:metric:user";
+
+ private CounterUpdate monitoringInfoToCounterUpdate(MonitoringInfo
monitoringInfo) {
+long value =
monitoringInfo.getMetric().getCounterData().getInt64Value();
+String urn = monitoringInfo.getUrn();
+
+String type = monitoringInfo.getType();
+
+// todomigryz: run MonitoringInfo through validation process.
Review comment:
This requires PR mentioned below to go in.
Technically, I can copy data from another PR, but format of counters is
still being defined. It will cause more merging and resolving in both PRs.
I suggest we keep it as is for this PR and generalize/fix it in following PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172706)
Time Spent: 4h 40m (was: 4.5h)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172686&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172686
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 06/Dec/18 16:51
Start Date: 06/Dec/18 16:51
Worklog Time Spent: 10m
Work Description: swegner commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239525848
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -351,7 +398,115 @@ void updateProgress() {
}
}
-private void updateMetrics(BeamFnApi.Metrics metrics) {
+// Keeping this as static class for this iteration. Will extract to
separate file and generalize
+// when more counter types are added.
+// todomigryz: define counter transformer factory
+// that can provide respective counter transformer for different type of
counters.
+// (ie RowCountCounterTranformer, MSecCounterTransformer,
UserCounterTransformer, etc)
+private static class MonitoringInfoToCounterUpdateTransformer {
+
+ private final Map transformIdMapping;
+
+ public MonitoringInfoToCounterUpdateTransformer(
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+ }
+
+ // todo: search code for "beam:metrics"... and replace them with
relevant enums from
Review comment:
Should this be fixed before merging?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172686)
Time Spent: 4h 10m (was: 4h)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172690&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172690
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 06/Dec/18 16:51
Start Date: 06/Dec/18 16:51
Worklog Time Spent: 10m
Work Description: swegner commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239524179
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -307,10 +344,17 @@ void updateProgress() {
grpcWriteOperation.abortWait();
}
-BeamFnApi.Metrics metrics =
MoreFutures.get(bundleProcessOperation.getMetrics());
+//TODO: Replace getProcessBundleProgress with getMonitoringInfos when
Metrics is deprecated.
Review comment:
+1 to tag a JIRA. It's useful to have some additional context for when the
TODO inevitably gets stale and needs clarification to be actionable.
The JIRA doesn't have to be explicitly for doing this cleanup. It should
track when this TODO is ready to implement or obsolete. In this case, it would
make sense to have a JIRA for deprecating the legacy Metrics.
Google's public style guides [offer similar
advice](https://google.github.io/styleguide/cppguide.html#TODO_Comments) (we
don't follow Google styleguides in Beam, although the advice is useful):
> Use TODO comments for code that is temporary, a short-term solution, or
good-enough but not perfect.
>
> TODOs should include the string TODO in all caps, followed by the name,
e-mail address, bug ID, or other identifier of the person or issue with the
best context about the problem referenced by the TODO. The main purpose is to
have a consistent TODO that can be searched to find out how to get more details
upon request. A TODO is not a commitment that the person referenced will fix
the problem. Thus when you create a TODO with a name, it is almost always your
name that is given.
>
> `// TODO(k...@gmail.com): Use a "*" here for concatenation operator.`
> `// TODO(Zeke) change this to use relations.`
> `// TODO(bug 12345): remove the "Last visitors" feature`
>
> If your TODO is of the form "At a future date do something" make sure that
you either include a very specific date ("Fix by November 2005") or a very
specific event ("Remove this code when all clients can handle XML responses.").
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172690)
Time Spent: 4.5h (was: 4h 20m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172691&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172691
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 06/Dec/18 16:51
Start Date: 06/Dec/18 16:51
Worklog Time Spent: 10m
Work Description: swegner commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239518531
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkItemStatusClient.java
##
@@ -263,20 +263,26 @@ private synchronized WorkItemStatus
createStatusUpdate(boolean isFinal) {
return status;
}
+ // todomigryz this method should return List instead of
updating member variable
Review comment:
`todomigryz` -> `TODO(migryz)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172691)
Time Spent: 4.5h (was: 4h 20m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172688&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172688
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 06/Dec/18 16:51
Start Date: 06/Dec/18 16:51
Worklog Time Spent: 10m
Work Description: swegner commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239524339
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -307,10 +347,17 @@ void updateProgress() {
grpcWriteOperation.abortWait();
}
-BeamFnApi.Metrics metrics =
MoreFutures.get(bundleProcessOperation.getMetrics());
+//TODO: Replace getProcessBundleProgress with getMonitoringInfos when
Metrics is deprecated.
+ProcessBundleProgressResponse processBundleProgressResponse =
+MoreFutures.get(bundleProcessOperation.getProcessBundleProgress());
+updateMetrics(processBundleProgressResponse.getMonitoringInfosList());
-updateMetrics(metrics);
+// Supporting deprecated metrics until all supported runners are
migrated to using
+// MonitoringInfos
+Metrics metrics = processBundleProgressResponse.getMetrics();
+updateMetricsDeprecated(metrics);
+// todomigryz: utilize monitoringInfos here.
Review comment:
`todomigryz` -> `TODO(migryz)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172688)
Time Spent: 4.5h (was: 4h 20m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172685&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172685
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 06/Dec/18 16:51
Start Date: 06/Dec/18 16:51
Worklog Time Spent: 10m
Work Description: swegner commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239524498
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -351,7 +398,115 @@ void updateProgress() {
}
}
-private void updateMetrics(BeamFnApi.Metrics metrics) {
+// Keeping this as static class for this iteration. Will extract to
separate file and generalize
+// when more counter types are added.
+// todomigryz: define counter transformer factory
Review comment:
`todomigryz` -> `TODO(migryz)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172685)
Time Spent: 4h (was: 3h 50m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 4h
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172687&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172687
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 06/Dec/18 16:51
Start Date: 06/Dec/18 16:51
Worklog Time Spent: 10m
Work Description: swegner commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239527436
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -351,7 +398,115 @@ void updateProgress() {
}
}
-private void updateMetrics(BeamFnApi.Metrics metrics) {
+// Keeping this as static class for this iteration. Will extract to
separate file and generalize
+// when more counter types are added.
+// todomigryz: define counter transformer factory
+// that can provide respective counter transformer for different type of
counters.
+// (ie RowCountCounterTranformer, MSecCounterTransformer,
UserCounterTransformer, etc)
+private static class MonitoringInfoToCounterUpdateTransformer {
+
+ private final Map transformIdMapping;
+
+ public MonitoringInfoToCounterUpdateTransformer(
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+ }
+
+ // todo: search code for "beam:metrics"... and replace them with
relevant enums from
+ // proto after rebasing above https://github.com/apache/beam/pull/6799
that
+ // introduces relevant proto entries.
+ final String BEAM_METRICS_USER_PREFIX = "beam:metric:user";
+
+ private CounterUpdate monitoringInfoToCounterUpdate(MonitoringInfo
monitoringInfo) {
+long value =
monitoringInfo.getMetric().getCounterData().getInt64Value();
+String urn = monitoringInfo.getUrn();
+
+String type = monitoringInfo.getType();
+
+// todomigryz: run MonitoringInfo through validation process.
+// refer to https://github.com/apache/beam/pull/6799
+
+if (urn.startsWith(BEAM_METRICS_USER_PREFIX)) {
+ if (!type.equals("beam:metrics:sum_int_64")) {
+LOG.warn(
Review comment:
Are there scenarios where we expect invalid metric types which should be
ignored? If so, please comment when this might be the case.
If a correct implementation should always send valid data, it's best to fail
rather than be resilient such that it's easier to find and fix bugs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172687)
Time Spent: 4h 20m (was: 4h 10m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172684&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172684
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 06/Dec/18 16:51
Start Date: 06/Dec/18 16:51
Worklog Time Spent: 10m
Work Description: swegner commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239520301
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -139,30 +150,44 @@ private AutoCloseable
progressTrackerCloseable(ProgressTracker progressTracker)
*/
@Override
public Iterable extractMetricUpdates() {
+List result = progressTracker.extractCounterUpdates();
+if ((result != null) && (result.size() > 0)) {
+ return result;
+}
+
+// BeamFnApi.Metrics was deprecated and replaced with more flexible
MonitoringInfo.
+// However some of SDKs have implementations that utilize
BeamFnApi.Metrics and were not yet
+// migrated to using new approach.
+// Falling back to using deprecated approach until all officially
supported SDKs complete migration.
Review comment:
Is there a JIRA that tracks this migration?
More generally: is it possible to make this comment more actionable?
Something of the form: `once this condition is true (all SDKs are migrated), do
this action (remove this fallback)`. It should be straightforward for anybody
to check on the migration status and then delete the code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172684)
Time Spent: 3h 50m (was: 3h 40m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172689&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172689
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 06/Dec/18 16:51
Start Date: 06/Dec/18 16:51
Worklog Time Spent: 10m
Work Description: swegner commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239525981
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -351,7 +398,115 @@ void updateProgress() {
}
}
-private void updateMetrics(BeamFnApi.Metrics metrics) {
+// Keeping this as static class for this iteration. Will extract to
separate file and generalize
+// when more counter types are added.
+// todomigryz: define counter transformer factory
+// that can provide respective counter transformer for different type of
counters.
+// (ie RowCountCounterTranformer, MSecCounterTransformer,
UserCounterTransformer, etc)
+private static class MonitoringInfoToCounterUpdateTransformer {
+
+ private final Map transformIdMapping;
+
+ public MonitoringInfoToCounterUpdateTransformer(
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+ }
+
+ // todo: search code for "beam:metrics"... and replace them with
relevant enums from
+ // proto after rebasing above https://github.com/apache/beam/pull/6799
that
+ // introduces relevant proto entries.
+ final String BEAM_METRICS_USER_PREFIX = "beam:metric:user";
+
+ private CounterUpdate monitoringInfoToCounterUpdate(MonitoringInfo
monitoringInfo) {
+long value =
monitoringInfo.getMetric().getCounterData().getInt64Value();
+String urn = monitoringInfo.getUrn();
+
+String type = monitoringInfo.getType();
+
+// todomigryz: run MonitoringInfo through validation process.
Review comment:
Should this be fixed before merging?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172689)
Time Spent: 4.5h (was: 4h 20m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172692&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172692
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 06/Dec/18 16:51
Start Date: 06/Dec/18 16:51
Worklog Time Spent: 10m
Work Description: swegner commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239525644
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -351,7 +398,115 @@ void updateProgress() {
}
}
-private void updateMetrics(BeamFnApi.Metrics metrics) {
+// Keeping this as static class for this iteration. Will extract to
separate file and generalize
Review comment:
This first sentence comment seems to refer specifically to the PR. I suggest
removing it as the rest of the comment is sufficiently descriptive.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172692)
Time Spent: 4.5h (was: 4h 20m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172553&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172553
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 06/Dec/18 01:30
Start Date: 06/Dec/18 01:30
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239301167
##
File path:
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutorTest.java
##
@@ -423,6 +434,187 @@ public void close() {}
contains(new
CounterHamcrestMatchers.CounterUpdateIntegerValueMatcher(finalCounterValue)));
}
+ @Test(timeout = ReadOperation.DEFAULT_PROGRESS_UPDATE_PERIOD_MS * 10)
+ public void
testExtractCounterUpdatesReturnsValidProgressTrackerCounterUpdatesIfPresent()
+ throws Exception {
+final String stepName = "fakeStepNameWithUserMetrics";
+final String namespace = "sdk/whatever";
+final String name = "someCounter";
+final int counterValue = 42;
+final int finalCounterValue = 77;
+final CountDownLatch progressSentLatch = new CountDownLatch(1);
+final CountDownLatch processBundleLatch = new CountDownLatch(1);
+
+final BeamFnApi.Metrics.User.MetricName metricName =
+BeamFnApi.Metrics.User.MetricName.newBuilder()
+.setNamespace(namespace)
+.setName(name)
+.build();
+
+final BeamFnApi.Metrics deprecatedMetrics =
+BeamFnApi.Metrics.newBuilder()
+.putPtransforms(GRPC_READ_ID, FAKE_ELEMENT_COUNT_METRICS)
+.putPtransforms(
+stepName,
+BeamFnApi.Metrics.PTransform.newBuilder()
+.addUser(
+BeamFnApi.Metrics.User.newBuilder()
+.setMetricName(metricName)
+.setCounterData(
+BeamFnApi.Metrics.User.CounterData.newBuilder()
+.setValue(finalCounterValue)))
+.build())
+.build();
+
+final int expectedCounterValue = 5;
+final BeamFnApi.MonitoringInfo expectedMonitoringInfo =
+BeamFnApi.MonitoringInfo.newBuilder()
+.setUrn("beam:metric:user:ExpectedCounter")
+.setType("beam:metrics:sum_int_64")
+.putLabels("PTRANSFORM", "ExpectedPTransform")
+.setMetric(
+BeamFnApi.Metric.newBuilder()
+.setCounterData(
+BeamFnApi.CounterData.newBuilder()
+.setInt64Value(expectedCounterValue)
+.build())
+.build())
+.build();
+
+InstructionRequestHandler instructionRequestHandler =
+new InstructionRequestHandler() {
+ @Override
+ public CompletionStage
handle(InstructionRequest request) {
+switch (request.getRequestCase()) {
+ case REGISTER:
+return
CompletableFuture.completedFuture(responseFor(request).build());
+ case PROCESS_BUNDLE:
+return MoreFutures.supplyAsync(
+() -> {
+ processBundleLatch.await();
+ return responseFor(request)
+ .setProcessBundle(
+ BeamFnApi.ProcessBundleResponse.newBuilder()
+ .setMetrics(deprecatedMetrics)
+ .addMonitoringInfos(expectedMonitoringInfo))
+ .build();
+});
+ case PROCESS_BUNDLE_PROGRESS:
+progressSentLatch.countDown();
+return CompletableFuture.completedFuture(
+responseFor(request)
+.setProcessBundleProgress(
+
BeamFnApi.ProcessBundleProgressResponse.newBuilder()
+.setMetrics(deprecatedMetrics)
+.addMonitoringInfos(expectedMonitoringInfo))
+.build());
+ default:
+throw new RuntimeException("Reached unexpected code path");
+}
+ }
+
+ @Override
+ public void close() {}
+};
+
+Map stepContextMap = new HashMap<>();
+NameContext nc =
+new NameContext() {
+ @Nullable
+ @Override
+ public String stageName() {
+return "ExpectedStage";
+ }
+
+ @Nullable
+ @Override
+ public String originalName() {
+return "Ex
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172552&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172552
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 06/Dec/18 01:30
Start Date: 06/Dec/18 01:30
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239301128
##
File path:
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutorTest.java
##
@@ -423,6 +434,187 @@ public void close() {}
contains(new
CounterHamcrestMatchers.CounterUpdateIntegerValueMatcher(finalCounterValue)));
}
+ @Test(timeout = ReadOperation.DEFAULT_PROGRESS_UPDATE_PERIOD_MS * 10)
+ public void
testExtractCounterUpdatesReturnsValidProgressTrackerCounterUpdatesIfPresent()
+ throws Exception {
+final String stepName = "fakeStepNameWithUserMetrics";
+final String namespace = "sdk/whatever";
+final String name = "someCounter";
+final int counterValue = 42;
+final int finalCounterValue = 77;
+final CountDownLatch progressSentLatch = new CountDownLatch(1);
+final CountDownLatch processBundleLatch = new CountDownLatch(1);
+
+final BeamFnApi.Metrics.User.MetricName metricName =
+BeamFnApi.Metrics.User.MetricName.newBuilder()
+.setNamespace(namespace)
+.setName(name)
+.build();
+
+final BeamFnApi.Metrics deprecatedMetrics =
+BeamFnApi.Metrics.newBuilder()
+.putPtransforms(GRPC_READ_ID, FAKE_ELEMENT_COUNT_METRICS)
+.putPtransforms(
+stepName,
+BeamFnApi.Metrics.PTransform.newBuilder()
+.addUser(
+BeamFnApi.Metrics.User.newBuilder()
+.setMetricName(metricName)
+.setCounterData(
+BeamFnApi.Metrics.User.CounterData.newBuilder()
+.setValue(finalCounterValue)))
+.build())
+.build();
+
+final int expectedCounterValue = 5;
+final BeamFnApi.MonitoringInfo expectedMonitoringInfo =
+BeamFnApi.MonitoringInfo.newBuilder()
+.setUrn("beam:metric:user:ExpectedCounter")
+.setType("beam:metrics:sum_int_64")
+.putLabels("PTRANSFORM", "ExpectedPTransform")
+.setMetric(
+BeamFnApi.Metric.newBuilder()
+.setCounterData(
+BeamFnApi.CounterData.newBuilder()
+.setInt64Value(expectedCounterValue)
+.build())
+.build())
+.build();
+
+InstructionRequestHandler instructionRequestHandler =
+new InstructionRequestHandler() {
+ @Override
+ public CompletionStage
handle(InstructionRequest request) {
+switch (request.getRequestCase()) {
+ case REGISTER:
+return
CompletableFuture.completedFuture(responseFor(request).build());
+ case PROCESS_BUNDLE:
+return MoreFutures.supplyAsync(
+() -> {
+ processBundleLatch.await();
+ return responseFor(request)
+ .setProcessBundle(
+ BeamFnApi.ProcessBundleResponse.newBuilder()
+ .setMetrics(deprecatedMetrics)
+ .addMonitoringInfos(expectedMonitoringInfo))
+ .build();
+});
+ case PROCESS_BUNDLE_PROGRESS:
+progressSentLatch.countDown();
+return CompletableFuture.completedFuture(
+responseFor(request)
+.setProcessBundleProgress(
+
BeamFnApi.ProcessBundleProgressResponse.newBuilder()
+.setMetrics(deprecatedMetrics)
+.addMonitoringInfos(expectedMonitoringInfo))
+.build());
+ default:
+throw new RuntimeException("Reached unexpected code path");
+}
+ }
+
+ @Override
+ public void close() {}
+};
+
+Map stepContextMap = new HashMap<>();
+NameContext nc =
+new NameContext() {
Review comment:
Do you need to override this and make a new class? Or is it possible to just
instantiate an existing class and passing in these strings?
This is an aut
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172530&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172530
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 06/Dec/18 00:03
Start Date: 06/Dec/18 00:03
Worklog Time Spent: 10m
Work Description: Ardagan commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239285290
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -307,10 +344,17 @@ void updateProgress() {
grpcWriteOperation.abortWait();
}
-BeamFnApi.Metrics metrics =
MoreFutures.get(bundleProcessOperation.getMetrics());
+//TODO: Replace getProcessBundleProgress with getMonitoringInfos when
Metrics is deprecated.
Review comment:
I was thinking on the proper way here and assumed that todo that explains
when the code gets deprecated was the best approach.
1. Creating JIRA that looks like "Cleanup Dataflow Runner Harness" when
Metrics are deprecated. Doesn't cover proper workload. So it is not the best
idea in my opinion.
2. Assigning username is not beneficial as well, since we do not know when
will we completely deprecate Metrics. So it doesn't really differ from regular
todo.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172530)
Time Spent: 3h 20m (was: 3h 10m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172529&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172529
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 06/Dec/18 00:02
Start Date: 06/Dec/18 00:02
Worklog Time Spent: 10m
Work Description: Ardagan commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239285290
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -307,10 +344,17 @@ void updateProgress() {
grpcWriteOperation.abortWait();
}
-BeamFnApi.Metrics metrics =
MoreFutures.get(bundleProcessOperation.getMetrics());
+//TODO: Replace getProcessBundleProgress with getMonitoringInfos when
Metrics is deprecated.
Review comment:
I was thinking on the proper way here and assumed that todo that clears up
when the code gets deprecated was the best approach.
1. Creating JIRA that looks like "Cleanup Dataflow Runner Harness" when
Metrics are deprecated. Doesn't cover proper workload. So it is not the best
idea in my opinion.
2. Assigning username is not beneficial as well, since we do not know when
will we completely deprecate Metrics. So it doesn't really differ from regular
todo.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172529)
Time Spent: 3h 10m (was: 3h)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172510&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172510
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 05/Dec/18 22:52
Start Date: 05/Dec/18 22:52
Worklog Time Spent: 10m
Work Description: Ardagan commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239268924
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -351,7 +398,106 @@ void updateProgress() {
}
}
-private void updateMetrics(BeamFnApi.Metrics metrics) {
+// Keeping this as static class for this iteration. Will extract to
separate file and generalize
+// when more counter types are added.
+// todomigryz: define counter transformer factory
+// that can provide respective counter transformer for different type of
counters.
+// (ie RowCountCounterTranformer, MSecCounterTransformer,
UserCounterTransformer, etc)
+private static class MonitoringInfoToCounterUpdateTransformer {
+
+ private final Map transformIdMapping;
+
+ public MonitoringInfoToCounterUpdateTransformer(
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+ }
+
+ // todo: search code for "beam:metrics"... and replace them with
relevant enums from
+ // proto after rebasing above https://github.com/apache/beam/pull/6799
that
+ // introduces relevant proto entries.
+ final String BEAM_METRICS_USER_PREFIX = "beam:metric:user";
+
+ private CounterUpdate monitoringInfoToCounterUpdate(MonitoringInfo
monitoringInfo) {
+long value =
monitoringInfo.getMetric().getCounterData().getInt64Value();
+String urn = monitoringInfo.getUrn();
+
+String type = monitoringInfo.getType();
+
+// todo: run MonitoringInfo through validation process.
+// refer to https://github.com/apache/beam/pull/6799
+
+if (urn.startsWith(BEAM_METRICS_USER_PREFIX)) {
+ if (!type.equals("beam:metrics:sum_int_64")) {
+return null;
+ }
+
+ final String ptransform =
monitoringInfo.getLabelsMap().get("PTRANSFORM");
+ if (ptransform == null) {
+return null;
+ }
+
+ DataflowStepContext stepContext = transformIdMapping.get(ptransform);
+
+ CounterStructuredNameAndMetadata name = new
CounterStructuredNameAndMetadata();
+
+ String nameWithNamespace =
+ monitoringInfo
+ .getUrn()
+ .substring(BEAM_METRICS_USER_PREFIX.length())
+ .replace("^:", "");
+
+ final int lastColonIndex = nameWithNamespace.lastIndexOf(':');
+ String counterName = nameWithNamespace.substring(lastColonIndex + 1);
+ String counterNamespace = nameWithNamespace.substring(0,
lastColonIndex);
+
+ name.setName(
+ new CounterStructuredName()
+ .setOrigin(Origin.USER.toString())
+ // Workaround for bug in python sdk that missed colon
after ...metric:user.
+ .setName(counterName)
+ .setOriginalStepName(
+ stepContext == null ? null :
stepContext.getNameContext().originalName())
+ .setExecutionStepName(
+ stepContext == null ? null :
stepContext.getNameContext().systemName())
+ .setOriginNamespace(counterNamespace))
+ .setMetadata(new CounterMetadata().setKind("SUM"));
+
+ return new CounterUpdate()
Review comment:
That is a valid point. But I'd prefer to implement this in next PR when I
add other type of counters.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172510)
Time Spent: 3h (was: 2h 50m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Pr
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172502&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172502
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 05/Dec/18 22:04
Start Date: 05/Dec/18 22:04
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r238894508
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -351,7 +395,104 @@ void updateProgress() {
}
}
-private void updateMetrics(BeamFnApi.Metrics metrics) {
+// todomigryz: define counter transformer factory (read wiki for proper
name)
+// that can provide respective counter transformer for different type of
counters.
+// (ie RowCountCounterTranformer, MSecCounterTransformer,
UserCounterTransformer, etc)
+private static class MonitoringInfoToCounterUpdateTransformer {
+
+ private final Map transformIdMapping;
+
+ public MonitoringInfoToCounterUpdateTransformer(
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+ }
+
+ // todo: search code for "beam:metrics"... and replace them with
relevant enums from
+ // proto after rebasing above https://github.com/apache/beam/pull/6799
that
+ // introduces relevant proto entries.
+ final String BEAM_METRICS_USER_PREFIX = "beam:metric:user";
+
+ private CounterUpdate monitoringInfoToCounterUpdate(MonitoringInfo
monitoringInfo) {
+long value =
monitoringInfo.getMetric().getCounterData().getInt64Value();
+String urn = monitoringInfo.getUrn();
+
+String type = monitoringInfo.getType();
+
+// todo: run MonitoringInfo through validation process.
+// refer to https://github.com/apache/beam/pull/6799
+
+if (urn.startsWith(BEAM_METRICS_USER_PREFIX)) {
+ if (!type.equals("beam:metrics:sum_int_64")) {
+return null;
+ }
+
+ final String ptransform =
monitoringInfo.getLabelsMap().get("PTRANSFORM");
+ if (ptransform == null) {
+return null;
+ }
+
+ DataflowStepContext stepContext = transformIdMapping.get(ptransform);
+
+ CounterStructuredNameAndMetadata name = new
CounterStructuredNameAndMetadata();
+
+ String nameWithNamespace =
+ monitoringInfo
+ .getUrn()
+ .substring(BEAM_METRICS_USER_PREFIX.length())
+ .replace("^:", "");
+
+ final int lastColonIndex = nameWithNamespace.lastIndexOf(':');
+ String counterName = nameWithNamespace.substring(lastColonIndex + 1);
+ String counterNamespace = nameWithNamespace.substring(0,
lastColonIndex);
+
+ name.setName(
+ new CounterStructuredName()
+ .setOrigin(Origin.USER.toString())
+ // Workaround for bug in python sdk that missed colon
after ...metric:user.
+ .setName(counterName)
+ .setOriginalStepName(
+ stepContext == null ? null :
stepContext.getNameContext().originalName())
Review comment:
I don't think we should ever set this to null, if we are unable to mape to a
stepContext we should drop the metric and log error/warning
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172502)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172491&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172491
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 05/Dec/18 22:04
Start Date: 05/Dec/18 22:04
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r238894238
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -351,7 +395,104 @@ void updateProgress() {
}
}
-private void updateMetrics(BeamFnApi.Metrics metrics) {
+// todomigryz: define counter transformer factory (read wiki for proper
name)
+// that can provide respective counter transformer for different type of
counters.
+// (ie RowCountCounterTranformer, MSecCounterTransformer,
UserCounterTransformer, etc)
+private static class MonitoringInfoToCounterUpdateTransformer {
+
+ private final Map transformIdMapping;
+
+ public MonitoringInfoToCounterUpdateTransformer(
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+ }
+
+ // todo: search code for "beam:metrics"... and replace them with
relevant enums from
+ // proto after rebasing above https://github.com/apache/beam/pull/6799
that
+ // introduces relevant proto entries.
+ final String BEAM_METRICS_USER_PREFIX = "beam:metric:user";
+
+ private CounterUpdate monitoringInfoToCounterUpdate(MonitoringInfo
monitoringInfo) {
+long value =
monitoringInfo.getMetric().getCounterData().getInt64Value();
+String urn = monitoringInfo.getUrn();
+
+String type = monitoringInfo.getType();
+
+// todo: run MonitoringInfo through validation process.
+// refer to https://github.com/apache/beam/pull/6799
+
+if (urn.startsWith(BEAM_METRICS_USER_PREFIX)) {
Review comment:
Architectural suggestion would be to use polymorphism instead of a switch
statement,
Make a class/object responsible for each URN.
Though I think much of it can be common, like label extraction/conversion.
Let's come up with something that will support some default behaviour and allow
you to override it with specific URN needs
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172491)
Time Spent: 1.5h (was: 1h 20m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172498&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172498
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 05/Dec/18 22:04
Start Date: 05/Dec/18 22:04
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239254784
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -351,7 +398,106 @@ void updateProgress() {
}
}
-private void updateMetrics(BeamFnApi.Metrics metrics) {
+// Keeping this as static class for this iteration. Will extract to
separate file and generalize
+// when more counter types are added.
+// todomigryz: define counter transformer factory
+// that can provide respective counter transformer for different type of
counters.
+// (ie RowCountCounterTranformer, MSecCounterTransformer,
UserCounterTransformer, etc)
+private static class MonitoringInfoToCounterUpdateTransformer {
+
+ private final Map transformIdMapping;
+
+ public MonitoringInfoToCounterUpdateTransformer(
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+ }
+
+ // todo: search code for "beam:metrics"... and replace them with
relevant enums from
+ // proto after rebasing above https://github.com/apache/beam/pull/6799
that
+ // introduces relevant proto entries.
+ final String BEAM_METRICS_USER_PREFIX = "beam:metric:user";
+
+ private CounterUpdate monitoringInfoToCounterUpdate(MonitoringInfo
monitoringInfo) {
+long value =
monitoringInfo.getMetric().getCounterData().getInt64Value();
+String urn = monitoringInfo.getUrn();
+
+String type = monitoringInfo.getType();
+
+// todo: run MonitoringInfo through validation process.
+// refer to https://github.com/apache/beam/pull/6799
+
+if (urn.startsWith(BEAM_METRICS_USER_PREFIX)) {
+ if (!type.equals("beam:metrics:sum_int_64")) {
+return null;
+ }
+
+ final String ptransform =
monitoringInfo.getLabelsMap().get("PTRANSFORM");
+ if (ptransform == null) {
+return null;
+ }
+
+ DataflowStepContext stepContext = transformIdMapping.get(ptransform);
+
+ CounterStructuredNameAndMetadata name = new
CounterStructuredNameAndMetadata();
+
+ String nameWithNamespace =
+ monitoringInfo
+ .getUrn()
+ .substring(BEAM_METRICS_USER_PREFIX.length())
+ .replace("^:", "");
+
+ final int lastColonIndex = nameWithNamespace.lastIndexOf(':');
+ String counterName = nameWithNamespace.substring(lastColonIndex + 1);
+ String counterNamespace = nameWithNamespace.substring(0,
lastColonIndex);
+
+ name.setName(
+ new CounterStructuredName()
+ .setOrigin(Origin.USER.toString())
+ // Workaround for bug in python sdk that missed colon
after ...metric:user.
+ .setName(counterName)
+ .setOriginalStepName(
+ stepContext == null ? null :
stepContext.getNameContext().originalName())
+ .setExecutionStepName(
+ stepContext == null ? null :
stepContext.getNameContext().systemName())
+ .setOriginNamespace(counterNamespace))
+ .setMetadata(new CounterMetadata().setKind("SUM"));
+
+ return new CounterUpdate()
+ .setStructuredNameAndMetadata(name)
+ .setCumulative(false)
+
.setInteger(DataflowCounterUpdateExtractor.longToSplitInt(value));
+}
+return null;
+ }
+}
+
+private void updateMetrics(List monitoringInfos) {
Review comment:
please add docstrings to all of these. There are many methods with too
similar names.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172498)
Time Spent: 2h 20m (was: 2h 10m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-618
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172496&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172496
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 05/Dec/18 22:04
Start Date: 05/Dec/18 22:04
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239247446
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -307,10 +347,17 @@ void updateProgress() {
grpcWriteOperation.abortWait();
}
-BeamFnApi.Metrics metrics =
MoreFutures.get(bundleProcessOperation.getMetrics());
+//TODO: Replace getProcessBundleProgress with getMonitoringInfos when
Metrics is deprecated.
+ProcessBundleProgressResponse processBundleProgressResponse =
+MoreFutures.get(bundleProcessOperation.getProcessBundleProgress());
+updateMetrics(processBundleProgressResponse.getMonitoringInfosList());
-updateMetrics(metrics);
+// Supporting deprecated metrics until all supported runners are
migrated to using
+// MonitoringInfos
+Metrics metrics = processBundleProgressResponse.getMetrics();
+updateMetricsDeprecated(metrics);
+// TODO: change this to utilize MonitroingInfos
Review comment:
please add username and/or jira number
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172496)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172493&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172493
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 05/Dec/18 22:04
Start Date: 05/Dec/18 22:04
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239240671
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkItemStatusClient.java
##
@@ -263,20 +263,35 @@ private synchronized WorkItemStatus
createStatusUpdate(boolean isFinal) {
return status;
}
+ // todo this method should return List instead of setting it
to WorkitemStatus
Review comment:
Please add a username and/or jira
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172493)
Time Spent: 1h 50m (was: 1h 40m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172504&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172504
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 05/Dec/18 22:07
Start Date: 05/Dec/18 22:07
Worklog Time Spent: 10m
Work Description: Ardagan commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239256276
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -351,7 +395,104 @@ void updateProgress() {
}
}
-private void updateMetrics(BeamFnApi.Metrics metrics) {
+// todomigryz: define counter transformer factory (read wiki for proper
name)
+// that can provide respective counter transformer for different type of
counters.
+// (ie RowCountCounterTranformer, MSecCounterTransformer,
UserCounterTransformer, etc)
+private static class MonitoringInfoToCounterUpdateTransformer {
+
+ private final Map transformIdMapping;
+
+ public MonitoringInfoToCounterUpdateTransformer(
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+ }
+
+ // todo: search code for "beam:metrics"... and replace them with
relevant enums from
+ // proto after rebasing above https://github.com/apache/beam/pull/6799
that
+ // introduces relevant proto entries.
+ final String BEAM_METRICS_USER_PREFIX = "beam:metric:user";
+
+ private CounterUpdate monitoringInfoToCounterUpdate(MonitoringInfo
monitoringInfo) {
+long value =
monitoringInfo.getMetric().getCounterData().getInt64Value();
+String urn = monitoringInfo.getUrn();
+
+String type = monitoringInfo.getType();
+
+// todo: run MonitoringInfo through validation process.
+// refer to https://github.com/apache/beam/pull/6799
+
+if (urn.startsWith(BEAM_METRICS_USER_PREFIX)) {
Review comment:
Agree with this one.
I decided to keep it this way for now. Will implement polymorphism when
adding support for another counter type.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172504)
Time Spent: 2h 50m (was: 2h 40m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172499&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172499
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 05/Dec/18 22:04
Start Date: 05/Dec/18 22:04
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239253828
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -351,7 +398,106 @@ void updateProgress() {
}
}
-private void updateMetrics(BeamFnApi.Metrics metrics) {
+// Keeping this as static class for this iteration. Will extract to
separate file and generalize
+// when more counter types are added.
+// todomigryz: define counter transformer factory
+// that can provide respective counter transformer for different type of
counters.
+// (ie RowCountCounterTranformer, MSecCounterTransformer,
UserCounterTransformer, etc)
+private static class MonitoringInfoToCounterUpdateTransformer {
+
+ private final Map transformIdMapping;
+
+ public MonitoringInfoToCounterUpdateTransformer(
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+ }
+
+ // todo: search code for "beam:metrics"... and replace them with
relevant enums from
+ // proto after rebasing above https://github.com/apache/beam/pull/6799
that
+ // introduces relevant proto entries.
+ final String BEAM_METRICS_USER_PREFIX = "beam:metric:user";
+
+ private CounterUpdate monitoringInfoToCounterUpdate(MonitoringInfo
monitoringInfo) {
+long value =
monitoringInfo.getMetric().getCounterData().getInt64Value();
+String urn = monitoringInfo.getUrn();
+
+String type = monitoringInfo.getType();
+
+// todo: run MonitoringInfo through validation process.
+// refer to https://github.com/apache/beam/pull/6799
+
+if (urn.startsWith(BEAM_METRICS_USER_PREFIX)) {
+ if (!type.equals("beam:metrics:sum_int_64")) {
+return null;
+ }
+
+ final String ptransform =
monitoringInfo.getLabelsMap().get("PTRANSFORM");
+ if (ptransform == null) {
+return null;
+ }
+
+ DataflowStepContext stepContext = transformIdMapping.get(ptransform);
+
+ CounterStructuredNameAndMetadata name = new
CounterStructuredNameAndMetadata();
+
+ String nameWithNamespace =
Review comment:
Please put this in a helper method for readability, make a method to parse
the string
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172499)
Time Spent: 2h 20m (was: 2h 10m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172490&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172490
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 05/Dec/18 22:04
Start Date: 05/Dec/18 22:04
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r238893122
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -307,10 +344,17 @@ void updateProgress() {
grpcWriteOperation.abortWait();
}
-BeamFnApi.Metrics metrics =
MoreFutures.get(bundleProcessOperation.getMetrics());
+//TODO: Replace getProcessBundleProgress with getMonitoringInfos when
Metrics is deprecated.
Review comment:
please add username and/or jira number
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172490)
Time Spent: 1h 20m (was: 1h 10m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172494&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172494
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 05/Dec/18 22:04
Start Date: 05/Dec/18 22:04
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239241898
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkItemStatusClient.java
##
@@ -263,20 +263,35 @@ private synchronized WorkItemStatus
createStatusUpdate(boolean isFinal) {
return status;
}
+ // todo this method should return List instead of setting it
to WorkitemStatus
@VisibleForTesting
synchronized void populateCounterUpdates(WorkItemStatus status) {
if (worker == null) {
return;
}
boolean isFinalUpdate = Boolean.TRUE.equals(status.getCompleted());
-ImmutableList.Builder counterUpdatesBuilder =
ImmutableList.builder();
-counterUpdatesBuilder.addAll(extractCounters(worker.getOutputCounters()));
-counterUpdatesBuilder.addAll(extractMetrics(isFinalUpdate));
-counterUpdatesBuilder.addAll(extractMsecCounters(isFinalUpdate));
-counterUpdatesBuilder.addAll(worker.extractMetricUpdates());
-ImmutableList counterUpdates =
counterUpdatesBuilder.build();
+ImmutableList.Builder counterUpdatesListBuilder =
ImmutableList.builder();
+Iterable newCounterUpdates;
+
+// Output counters
+newCounterUpdates = extractCounters(worker.getOutputCounters());
+counterUpdatesListBuilder.addAll(newCounterUpdates);
+
+// User metrics reported in Worker
+newCounterUpdates = extractMetrics(isFinalUpdate);
Review comment:
Please revert to the previous style, there isn't much value to expanding
these to two lines each.
thanks for the code comments though
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172494)
Time Spent: 2h (was: 1h 50m)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172503&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172503
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 05/Dec/18 22:04
Start Date: 05/Dec/18 22:04
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239245981
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -307,10 +347,17 @@ void updateProgress() {
grpcWriteOperation.abortWait();
}
-BeamFnApi.Metrics metrics =
MoreFutures.get(bundleProcessOperation.getMetrics());
+//TODO: Replace getProcessBundleProgress with getMonitoringInfos when
Metrics is deprecated.
+ProcessBundleProgressResponse processBundleProgressResponse =
+MoreFutures.get(bundleProcessOperation.getProcessBundleProgress());
+updateMetrics(processBundleProgressResponse.getMonitoringInfosList());
-updateMetrics(metrics);
+// Supporting deprecated metrics until all supported runners are
migrated to using
+// MonitoringInfos
+Metrics metrics = processBundleProgressResponse.getMetrics();
+updateMetricsDeprecated(metrics);
+// TODO: change this to utilize MonitroingInfos
Review comment:
sp: MonitroingInfos
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172503)
Time Spent: 2h 40m (was: 2.5h)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172497&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172497
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 05/Dec/18 22:04
Start Date: 05/Dec/18 22:04
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239252005
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -351,7 +398,106 @@ void updateProgress() {
}
}
-private void updateMetrics(BeamFnApi.Metrics metrics) {
+// Keeping this as static class for this iteration. Will extract to
separate file and generalize
+// when more counter types are added.
+// todomigryz: define counter transformer factory
+// that can provide respective counter transformer for different type of
counters.
+// (ie RowCountCounterTranformer, MSecCounterTransformer,
UserCounterTransformer, etc)
+private static class MonitoringInfoToCounterUpdateTransformer {
+
+ private final Map transformIdMapping;
+
+ public MonitoringInfoToCounterUpdateTransformer(
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+ }
+
+ // todo: search code for "beam:metrics"... and replace them with
relevant enums from
+ // proto after rebasing above https://github.com/apache/beam/pull/6799
that
+ // introduces relevant proto entries.
+ final String BEAM_METRICS_USER_PREFIX = "beam:metric:user";
+
+ private CounterUpdate monitoringInfoToCounterUpdate(MonitoringInfo
monitoringInfo) {
+long value =
monitoringInfo.getMetric().getCounterData().getInt64Value();
+String urn = monitoringInfo.getUrn();
+
+String type = monitoringInfo.getType();
+
+// todo: run MonitoringInfo through validation process.
Review comment:
Add username and/or jira number
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172497)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172489&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172489
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 05/Dec/18 22:04
Start Date: 05/Dec/18 22:04
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r238892994
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -177,21 +199,24 @@ public ReadOperation getReadOperation() throws Exception
{
throw new IllegalStateException(String.format("ReadOperation not found in
%s", operations));
}
- private static interface ProgressTracker {
+ private interface ProgressTracker {
@Nullable
-public NativeReader.Progress getWorkerProgress() throws Exception;
+public Progress getWorkerProgress() throws Exception;
/**
* Returns an metric updates accumulated since the last call to {@link
#extractMetricUpdates()}.
*/
+@Deprecated
public MetricUpdates extractMetricUpdates();
+public List extractCounterUpdates();
Review comment:
please add a comment describing what these are
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172489)
Time Spent: 1h 10m (was: 1h)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> New approach to report metrics in FnApi is to utilize MetricInfo structures.
> This approach is implemented in Python SDK and work is ongoing in Java SDK.
> This tasks includes plumbing User metrics reported via MetricInfos through
> Dataflow Java Runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172492&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172492
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 05/Dec/18 22:04
Start Date: 05/Dec/18 22:04
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239254570
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -351,7 +398,106 @@ void updateProgress() {
}
}
-private void updateMetrics(BeamFnApi.Metrics metrics) {
+// Keeping this as static class for this iteration. Will extract to
separate file and generalize
+// when more counter types are added.
+// todomigryz: define counter transformer factory
+// that can provide respective counter transformer for different type of
counters.
+// (ie RowCountCounterTranformer, MSecCounterTransformer,
UserCounterTransformer, etc)
+private static class MonitoringInfoToCounterUpdateTransformer {
+
+ private final Map transformIdMapping;
+
+ public MonitoringInfoToCounterUpdateTransformer(
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+ }
+
+ // todo: search code for "beam:metrics"... and replace them with
relevant enums from
+ // proto after rebasing above https://github.com/apache/beam/pull/6799
that
+ // introduces relevant proto entries.
+ final String BEAM_METRICS_USER_PREFIX = "beam:metric:user";
+
+ private CounterUpdate monitoringInfoToCounterUpdate(MonitoringInfo
monitoringInfo) {
+long value =
monitoringInfo.getMetric().getCounterData().getInt64Value();
+String urn = monitoringInfo.getUrn();
+
+String type = monitoringInfo.getType();
+
+// todo: run MonitoringInfo through validation process.
+// refer to https://github.com/apache/beam/pull/6799
+
+if (urn.startsWith(BEAM_METRICS_USER_PREFIX)) {
+ if (!type.equals("beam:metrics:sum_int_64")) {
+return null;
+ }
+
+ final String ptransform =
monitoringInfo.getLabelsMap().get("PTRANSFORM");
+ if (ptransform == null) {
+return null;
+ }
+
+ DataflowStepContext stepContext = transformIdMapping.get(ptransform);
+
+ CounterStructuredNameAndMetadata name = new
CounterStructuredNameAndMetadata();
+
+ String nameWithNamespace =
+ monitoringInfo
+ .getUrn()
+ .substring(BEAM_METRICS_USER_PREFIX.length())
+ .replace("^:", "");
+
+ final int lastColonIndex = nameWithNamespace.lastIndexOf(':');
+ String counterName = nameWithNamespace.substring(lastColonIndex + 1);
+ String counterNamespace = nameWithNamespace.substring(0,
lastColonIndex);
+
+ name.setName(
+ new CounterStructuredName()
+ .setOrigin(Origin.USER.toString())
+ // Workaround for bug in python sdk that missed colon
after ...metric:user.
+ .setName(counterName)
+ .setOriginalStepName(
+ stepContext == null ? null :
stepContext.getNameContext().originalName())
+ .setExecutionStepName(
+ stepContext == null ? null :
stepContext.getNameContext().systemName())
+ .setOriginNamespace(counterNamespace))
+ .setMetadata(new CounterMetadata().setKind("SUM"));
+
+ return new CounterUpdate()
Review comment:
This is okay for now, since you are just unpacking the user counter metric.
but I want to see this share more code for different metrics later on.
Any label extraction can be shared for example.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172492)
Time Spent: 1h 40m (was: 1.5h)
> Utilize MetricInfo for reporting user metrics in Portable Dataflow Java
> Runner.
> ---
>
> Key: BEAM-6181
> URL: https://issues.apache.org/jira/browse/BEAM-6181
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execu
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172501&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172501
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 05/Dec/18 22:04
Start Date: 05/Dec/18 22:04
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239254686
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -351,7 +398,106 @@ void updateProgress() {
}
}
-private void updateMetrics(BeamFnApi.Metrics metrics) {
+// Keeping this as static class for this iteration. Will extract to
separate file and generalize
+// when more counter types are added.
+// todomigryz: define counter transformer factory
+// that can provide respective counter transformer for different type of
counters.
+// (ie RowCountCounterTranformer, MSecCounterTransformer,
UserCounterTransformer, etc)
+private static class MonitoringInfoToCounterUpdateTransformer {
+
+ private final Map transformIdMapping;
+
+ public MonitoringInfoToCounterUpdateTransformer(
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+ }
+
+ // todo: search code for "beam:metrics"... and replace them with
relevant enums from
+ // proto after rebasing above https://github.com/apache/beam/pull/6799
that
+ // introduces relevant proto entries.
+ final String BEAM_METRICS_USER_PREFIX = "beam:metric:user";
+
+ private CounterUpdate monitoringInfoToCounterUpdate(MonitoringInfo
monitoringInfo) {
+long value =
monitoringInfo.getMetric().getCounterData().getInt64Value();
+String urn = monitoringInfo.getUrn();
+
+String type = monitoringInfo.getType();
+
+// todo: run MonitoringInfo through validation process.
+// refer to https://github.com/apache/beam/pull/6799
+
+if (urn.startsWith(BEAM_METRICS_USER_PREFIX)) {
+ if (!type.equals("beam:metrics:sum_int_64")) {
+return null;
+ }
+
+ final String ptransform =
monitoringInfo.getLabelsMap().get("PTRANSFORM");
+ if (ptransform == null) {
+return null;
+ }
+
+ DataflowStepContext stepContext = transformIdMapping.get(ptransform);
+
+ CounterStructuredNameAndMetadata name = new
CounterStructuredNameAndMetadata();
+
+ String nameWithNamespace =
+ monitoringInfo
+ .getUrn()
+ .substring(BEAM_METRICS_USER_PREFIX.length())
+ .replace("^:", "");
+
+ final int lastColonIndex = nameWithNamespace.lastIndexOf(':');
+ String counterName = nameWithNamespace.substring(lastColonIndex + 1);
+ String counterNamespace = nameWithNamespace.substring(0,
lastColonIndex);
+
+ name.setName(
+ new CounterStructuredName()
+ .setOrigin(Origin.USER.toString())
+ // Workaround for bug in python sdk that missed colon
after ...metric:user.
+ .setName(counterName)
+ .setOriginalStepName(
+ stepContext == null ? null :
stepContext.getNameContext().originalName())
+ .setExecutionStepName(
+ stepContext == null ? null :
stepContext.getNameContext().systemName())
+ .setOriginNamespace(counterNamespace))
+ .setMetadata(new CounterMetadata().setKind("SUM"));
+
+ return new CounterUpdate()
+ .setStructuredNameAndMetadata(name)
+ .setCumulative(false)
+
.setInteger(DataflowCounterUpdateExtractor.longToSplitInt(value));
+}
+return null;
+ }
+}
+
+private void updateMetrics(List monitoringInfos) {
+ final MonitoringInfoToCounterUpdateTransformer
monitoringInfoToCounterUpdateTransformer =
+ new MonitoringInfoToCounterUpdateTransformer(
+ bundleProcessOperation.getPtransformIdToUserStepContext());
+ String miDump =
Review comment:
rm debug code
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172501)
Time Spent: 2.5h (was: 2h 20m)
> Utilize MetricInfo f
[jira] [Work logged] (BEAM-6181) Utilize MetricInfo for reporting user metrics in Portable Dataflow Java Runner.
[
https://issues.apache.org/jira/browse/BEAM-6181?focusedWorklogId=172495&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172495
]
ASF GitHub Bot logged work on BEAM-6181:
Author: ASF GitHub Bot
Created on: 05/Dec/18 22:04
Start Date: 05/Dec/18 22:04
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #7202:
[BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow
Runner.
URL: https://github.com/apache/beam/pull/7202#discussion_r239254849
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
##
@@ -351,7 +398,106 @@ void updateProgress() {
}
}
-private void updateMetrics(BeamFnApi.Metrics metrics) {
+// Keeping this as static class for this iteration. Will extract to
separate file and generalize
+// when more counter types are added.
+// todomigryz: define counter transformer factory
+// that can provide respective counter transformer for different type of
counters.
+// (ie RowCountCounterTranformer, MSecCounterTransformer,
UserCounterTransformer, etc)
+private static class MonitoringInfoToCounterUpdateTransformer {
+
+ private final Map transformIdMapping;
+
+ public MonitoringInfoToCounterUpdateTransformer(
+ final Map transformIdMapping) {
+this.transformIdMapping = transformIdMapping;
+ }
+
+ // todo: search code for "beam:metrics"... and replace them with
relevant enums from
+ // proto after rebasing above https://github.com/apache/beam/pull/6799
that
+ // introduces relevant proto entries.
+ final String BEAM_METRICS_USER_PREFIX = "beam:metric:user";
+
+ private CounterUpdate monitoringInfoToCounterUpdate(MonitoringInfo
monitoringInfo) {
+long value =
monitoringInfo.getMetric().getCounterData().getInt64Value();
+String urn = monitoringInfo.getUrn();
+
+String type = monitoringInfo.getType();
+
+// todo: run MonitoringInfo through validation process.
+// refer to https://github.com/apache/beam/pull/6799
+
+if (urn.startsWith(BEAM_METRICS_USER_PREFIX)) {
+ if (!type.equals("beam:metrics:sum_int_64")) {
+return null;
+ }
+
+ final String ptransform =
monitoringInfo.getLabelsMap().get("PTRANSFORM");
+ if (ptransform == null) {
+return null;
+ }
+
+ DataflowStepContext stepContext = transformIdMapping.get(ptransform);
+
+ CounterStructuredNameAndMetadata name = new
CounterStructuredNameAndMetadata();
+
+ String nameWithNamespace =
+ monitoringInfo
+ .getUrn()
+ .substring(BEAM_METRICS_USER_PREFIX.length())
+ .replace("^:", "");
+
+ final int lastColonIndex = nameWithNamespace.lastIndexOf(':');
+ String counterName = nameWithNamespace.substring(lastColonIndex + 1);
+ String counterNamespace = nameWithNamespace.substring(0,
lastColonIndex);
+
+ name.setName(
+ new CounterStructuredName()
+ .setOrigin(Origin.USER.toString())
+ // Workaround for bug in python sdk that missed colon
after ...metric:user.
+ .setName(counterName)
+ .setOriginalStepName(
+ stepContext == null ? null :
stepContext.getNameContext().originalName())
+ .setExecutionStepName(
+ stepContext == null ? null :
stepContext.getNameContext().systemName())
+ .setOriginNamespace(counterNamespace))
+ .setMetadata(new CounterMetadata().setKind("SUM"));
+
+ return new CounterUpdate()
+ .setStructuredNameAndMetadata(name)
+ .setCumulative(false)
+
.setInteger(DataflowCounterUpdateExtractor.longToSplitInt(value));
+}
+return null;
+ }
+}
+
+private void updateMetrics(List monitoringInfos) {
+ final MonitoringInfoToCounterUpdateTransformer
monitoringInfoToCounterUpdateTransformer =
+ new MonitoringInfoToCounterUpdateTransformer(
+ bundleProcessOperation.getPtransformIdToUserStepContext());
+ String miDump =
+
monitoringInfos.stream().map(Objects::toString).collect(Collectors.joining("\n"));
+
+ counterUpdates =
+ monitoringInfos
+ .stream()
+
.map(monitoringInfoToCounterUpdateTransformer::monitoringInfoToCounterUpdate)
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList());
+ return;
+}
+
+private String getStackTrace() {
Review comment:
rm debug code
---
