[jira] [Work logged] (BEAM-6431) Add ExecutionTime metrics to the Beam Java SDK
[ https://issues.apache.org/jira/browse/BEAM-6431?focusedWorklogId=194812=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194812 ] ASF GitHub Bot logged work on BEAM-6431: Author: ASF GitHub Bot Created on: 05/Feb/19 21:19 Start Date: 05/Feb/19 21:19 Worklog Time Spent: 10m Work Description: swegner commented on pull request #7676: [BEAM-6431] Implement Execution Time metrics start,process,finish in the Java SDK URL: https://github.com/apache/beam/pull/7676 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 194812) Time Spent: 3h 20m (was: 3h 10m) > Add ExecutionTime metrics to the Beam Java SDK > -- > > Key: BEAM-6431 > URL: https://issues.apache.org/jira/browse/BEAM-6431 > Project: Beam > Issue Type: New Feature > Components: java-fn-execution >Reporter: Alex Amato >Assignee: Alex Amato >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > This will be done by using the Dataflow worker's StateSampler code. I have > put together a refactoring plan > [here|https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#] > This will include estimating the processing time for the start, process and > finish bundle. The python SDK already has an implementation of this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6431) Add ExecutionTime metrics to the Beam Java SDK
[
https://issues.apache.org/jira/browse/BEAM-6431?focusedWorklogId=194747=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194747
]
ASF GitHub Bot logged work on BEAM-6431:
Author: ASF GitHub Bot
Created on: 05/Feb/19 19:00
Start Date: 05/Feb/19 19:00
Worklog Time Spent: 10m
Work Description: ajamato commented on pull request #7676: DO NOT SUBMIT
[BEAM-6431] Implement Execution Time metrics start,process,finish in the Java
SDK
URL: https://github.com/apache/beam/pull/7676#discussion_r254001725
##
File path:
runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
##
@@ -510,30 +514,33 @@ public void testMetrics() throws Exception {
ParDo.of(
new DoFn() {
private boolean emitted = false;
- Counter startCounter =
+ private Counter startCounter =
Metrics.counter(RemoteExecutionTest.class,
startUserCounterName);
@StartBundle
public void startBundle() throws InterruptedException {
+Thread.sleep(1000);
Review comment:
The one way we could remove these sleeps is by exposing the metrics during
the test, and polling until we see them. However, the metrics are only
available when the bundle finishes.
I filed BEAM-6597 with a TODO on how to do this and upgrade this test.
I don't want to go much lower than this, for fear of making this test
noisy/flakey.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 194747)
Time Spent: 3h 10m (was: 3h)
> Add ExecutionTime metrics to the Beam Java SDK
> --
>
> Key: BEAM-6431
> URL: https://issues.apache.org/jira/browse/BEAM-6431
> Project: Beam
> Issue Type: New Feature
> Components: java-fn-execution
>Reporter: Alex Amato
>Assignee: Alex Amato
>Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> This will be done by using the Dataflow worker's StateSampler code. I have
> put together a refactoring plan
> [here|https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#]
> This will include estimating the processing time for the start, process and
> finish bundle. The python SDK already has an implementation of this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6431) Add ExecutionTime metrics to the Beam Java SDK
[
https://issues.apache.org/jira/browse/BEAM-6431?focusedWorklogId=194716=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194716
]
ASF GitHub Bot logged work on BEAM-6431:
Author: ASF GitHub Bot
Created on: 05/Feb/19 18:06
Start Date: 05/Feb/19 18:06
Worklog Time Spent: 10m
Work Description: ajamato commented on pull request #7676: DO NOT SUBMIT
[BEAM-6431] Implement Execution Time metrics start,process,finish in the Java
SDK
URL: https://github.com/apache/beam/pull/7676#discussion_r253981147
##
File path:
runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MonitoringInfoMatchers.java
##
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.metrics;
+
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo;
+import org.hamcrest.Description;
+import org.hamcrest.TypeSafeMatcher;
+
+/** Matchers for {@link MonitoringInfo}. */
+public class MonitoringInfoMatchers {
+
+ /**
+ * Matches a {@link MonitoringInfo} with that has the set fields in the
provide MonitoringInfo.
+ *
+ * This is useful for tests which do not want to match the specific value
(execution times).
+ * Currently this will only check for URNs, labels, type URNs and
int64Values.
+ */
+ public static TypeSafeMatcher matchSetFields(final
MonitoringInfo mi) {
+return new TypeSafeMatcher() {
+
+ @Override
+ protected boolean matchesSafely(MonitoringInfo item) {
+// Match the URN.
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 194716)
Time Spent: 2h 50m (was: 2h 40m)
> Add ExecutionTime metrics to the Beam Java SDK
> --
>
> Key: BEAM-6431
> URL: https://issues.apache.org/jira/browse/BEAM-6431
> Project: Beam
> Issue Type: New Feature
> Components: java-fn-execution
>Reporter: Alex Amato
>Assignee: Alex Amato
>Priority: Major
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> This will be done by using the Dataflow worker's StateSampler code. I have
> put together a refactoring plan
> [here|https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#]
> This will include estimating the processing time for the start, process and
> finish bundle. The python SDK already has an implementation of this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6431) Add ExecutionTime metrics to the Beam Java SDK
[ https://issues.apache.org/jira/browse/BEAM-6431?focusedWorklogId=194717=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194717 ] ASF GitHub Bot logged work on BEAM-6431: Author: ASF GitHub Bot Created on: 05/Feb/19 18:06 Start Date: 05/Feb/19 18:06 Worklog Time Spent: 10m Work Description: ajamato commented on pull request #7676: DO NOT SUBMIT [BEAM-6431] Implement Execution Time metrics start,process,finish in the Java SDK URL: https://github.com/apache/beam/pull/7676#discussion_r253981356 ## File path: sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/PCollectionConsumerRegistry.java ## @@ -113,6 +134,12 @@ public PCollectionConsumerRegistry(MetricsContainerStepMap metricsContainerRegis return wrappedConsumer; } + /** @return Execution Time MonitoringInfos based on the tracked start or finish function. */ Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 194717) Time Spent: 3h (was: 2h 50m) > Add ExecutionTime metrics to the Beam Java SDK > -- > > Key: BEAM-6431 > URL: https://issues.apache.org/jira/browse/BEAM-6431 > Project: Beam > Issue Type: New Feature > Components: java-fn-execution >Reporter: Alex Amato >Assignee: Alex Amato >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > This will be done by using the Dataflow worker's StateSampler code. I have > put together a refactoring plan > [here|https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#] > This will include estimating the processing time for the start, process and > finish bundle. The python SDK already has an implementation of this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6431) Add ExecutionTime metrics to the Beam Java SDK
[ https://issues.apache.org/jira/browse/BEAM-6431?focusedWorklogId=194710=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194710 ] ASF GitHub Bot logged work on BEAM-6431: Author: ASF GitHub Bot Created on: 05/Feb/19 17:55 Start Date: 05/Feb/19 17:55 Worklog Time Spent: 10m Work Description: ajamato commented on pull request #7676: DO NOT SUBMIT [BEAM-6431] Implement Execution Time metrics start,process,finish in the Java SDK URL: https://github.com/apache/beam/pull/7676#discussion_r253976922 ## File path: runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/SimpleExecutionState.java ## @@ -0,0 +1,59 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.beam.runners.core.metrics; + +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; +import org.apache.beam.runners.core.metrics.ExecutionStateTracker.ExecutionState; + +/** + * Simple state class which collects the totalMillis spent in the state. Allows attaching an Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 194710) Time Spent: 2h 40m (was: 2.5h) > Add ExecutionTime metrics to the Beam Java SDK > -- > > Key: BEAM-6431 > URL: https://issues.apache.org/jira/browse/BEAM-6431 > Project: Beam > Issue Type: New Feature > Components: java-fn-execution >Reporter: Alex Amato >Assignee: Alex Amato >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > This will be done by using the Dataflow worker's StateSampler code. I have > put together a refactoring plan > [here|https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#] > This will include estimating the processing time for the start, process and > finish bundle. The python SDK already has an implementation of this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6431) Add ExecutionTime metrics to the Beam Java SDK
[ https://issues.apache.org/jira/browse/BEAM-6431?focusedWorklogId=194655=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194655 ] ASF GitHub Bot logged work on BEAM-6431: Author: ASF GitHub Bot Created on: 05/Feb/19 16:35 Start Date: 05/Feb/19 16:35 Worklog Time Spent: 10m Work Description: Ardagan commented on pull request #7676: DO NOT SUBMIT [BEAM-6431] Implement Execution Time metrics start,process,finish in the Java SDK URL: https://github.com/apache/beam/pull/7676#discussion_r253943246 ## File path: sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/PCollectionConsumerRegistry.java ## @@ -113,6 +134,12 @@ public PCollectionConsumerRegistry(MetricsContainerStepMap metricsContainerRegis return wrappedConsumer; } + /** @return Execution Time MonitoringInfos based on the tracked start or finish function. */ Review comment: duplicate comment This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 194655) > Add ExecutionTime metrics to the Beam Java SDK > -- > > Key: BEAM-6431 > URL: https://issues.apache.org/jira/browse/BEAM-6431 > Project: Beam > Issue Type: New Feature > Components: java-fn-execution >Reporter: Alex Amato >Assignee: Alex Amato >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > This will be done by using the Dataflow worker's StateSampler code. I have > put together a refactoring plan > [here|https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#] > This will include estimating the processing time for the start, process and > finish bundle. The python SDK already has an implementation of this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6431) Add ExecutionTime metrics to the Beam Java SDK
[
https://issues.apache.org/jira/browse/BEAM-6431?focusedWorklogId=194656=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194656
]
ASF GitHub Bot logged work on BEAM-6431:
Author: ASF GitHub Bot
Created on: 05/Feb/19 16:35
Start Date: 05/Feb/19 16:35
Worklog Time Spent: 10m
Work Description: Ardagan commented on pull request #7676: DO NOT SUBMIT
[BEAM-6431] Implement Execution Time metrics start,process,finish in the Java
SDK
URL: https://github.com/apache/beam/pull/7676#discussion_r253941309
##
File path:
runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
##
@@ -510,30 +514,33 @@ public void testMetrics() throws Exception {
ParDo.of(
new DoFn() {
private boolean emitted = false;
- Counter startCounter =
+ private Counter startCounter =
Metrics.counter(RemoteExecutionTest.class,
startUserCounterName);
@StartBundle
public void startBundle() throws InterruptedException {
+Thread.sleep(1000);
Review comment:
any way for us to use smaller values here, or avoid Thread.sleep at all?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 194656)
Time Spent: 2.5h (was: 2h 20m)
> Add ExecutionTime metrics to the Beam Java SDK
> --
>
> Key: BEAM-6431
> URL: https://issues.apache.org/jira/browse/BEAM-6431
> Project: Beam
> Issue Type: New Feature
> Components: java-fn-execution
>Reporter: Alex Amato
>Assignee: Alex Amato
>Priority: Major
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> This will be done by using the Dataflow worker's StateSampler code. I have
> put together a refactoring plan
> [here|https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#]
> This will include estimating the processing time for the start, process and
> finish bundle. The python SDK already has an implementation of this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6431) Add ExecutionTime metrics to the Beam Java SDK
[ https://issues.apache.org/jira/browse/BEAM-6431?focusedWorklogId=194653=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194653 ] ASF GitHub Bot logged work on BEAM-6431: Author: ASF GitHub Bot Created on: 05/Feb/19 16:35 Start Date: 05/Feb/19 16:35 Worklog Time Spent: 10m Work Description: Ardagan commented on pull request #7676: DO NOT SUBMIT [BEAM-6431] Implement Execution Time metrics start,process,finish in the Java SDK URL: https://github.com/apache/beam/pull/7676#discussion_r253937506 ## File path: runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/SimpleExecutionState.java ## @@ -0,0 +1,59 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.beam.runners.core.metrics; + +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; +import org.apache.beam.runners.core.metrics.ExecutionStateTracker.ExecutionState; + +/** + * Simple state class which collects the totalMillis spent in the state. Allows attaching an Review comment: Would be good to specify what it attaches key value labels to. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 194653) Time Spent: 2h 20m (was: 2h 10m) > Add ExecutionTime metrics to the Beam Java SDK > -- > > Key: BEAM-6431 > URL: https://issues.apache.org/jira/browse/BEAM-6431 > Project: Beam > Issue Type: New Feature > Components: java-fn-execution >Reporter: Alex Amato >Assignee: Alex Amato >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > This will be done by using the Dataflow worker's StateSampler code. I have > put together a refactoring plan > [here|https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#] > This will include estimating the processing time for the start, process and > finish bundle. The python SDK already has an implementation of this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6431) Add ExecutionTime metrics to the Beam Java SDK
[
https://issues.apache.org/jira/browse/BEAM-6431?focusedWorklogId=194654=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194654
]
ASF GitHub Bot logged work on BEAM-6431:
Author: ASF GitHub Bot
Created on: 05/Feb/19 16:35
Start Date: 05/Feb/19 16:35
Worklog Time Spent: 10m
Work Description: Ardagan commented on pull request #7676: DO NOT SUBMIT
[BEAM-6431] Implement Execution Time metrics start,process,finish in the Java
SDK
URL: https://github.com/apache/beam/pull/7676#discussion_r253939081
##
File path:
runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MonitoringInfoMatchers.java
##
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.metrics;
+
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo;
+import org.hamcrest.Description;
+import org.hamcrest.TypeSafeMatcher;
+
+/** Matchers for {@link MonitoringInfo}. */
+public class MonitoringInfoMatchers {
+
+ /**
+ * Matches a {@link MonitoringInfo} with that has the set fields in the
provide MonitoringInfo.
+ *
+ * This is useful for tests which do not want to match the specific value
(execution times).
+ * Currently this will only check for URNs, labels, type URNs and
int64Values.
+ */
+ public static TypeSafeMatcher matchSetFields(final
MonitoringInfo mi) {
+return new TypeSafeMatcher() {
+
+ @Override
+ protected boolean matchesSafely(MonitoringInfo item) {
+// Match the URN.
Review comment:
you can remove comments here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 194654)
Time Spent: 2h 20m (was: 2h 10m)
> Add ExecutionTime metrics to the Beam Java SDK
> --
>
> Key: BEAM-6431
> URL: https://issues.apache.org/jira/browse/BEAM-6431
> Project: Beam
> Issue Type: New Feature
> Components: java-fn-execution
>Reporter: Alex Amato
>Assignee: Alex Amato
>Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> This will be done by using the Dataflow worker's StateSampler code. I have
> put together a refactoring plan
> [here|https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#]
> This will include estimating the processing time for the start, process and
> finish bundle. The python SDK already has an implementation of this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6431) Add ExecutionTime metrics to the Beam Java SDK
[ https://issues.apache.org/jira/browse/BEAM-6431?focusedWorklogId=192481=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-192481 ] ASF GitHub Bot logged work on BEAM-6431: Author: ASF GitHub Bot Created on: 30/Jan/19 20:14 Start Date: 30/Jan/19 20:14 Worklog Time Spent: 10m Work Description: ajamato commented on pull request #7676: DO NOT SUBMIT [BEAM-6431] Implement Execution Time metrics start,process,finish in the Java SDK URL: https://github.com/apache/beam/pull/7676 [BEAM-6431] Implement Execution Time metrics start,process,finish in the Java SDK **Please** add a meaningful description for your change here Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking ---
[jira] [Work logged] (BEAM-6431) Add ExecutionTime metrics to the Beam Java SDK
[ https://issues.apache.org/jira/browse/BEAM-6431?focusedWorklogId=191854=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-191854 ] ASF GitHub Bot logged work on BEAM-6431: Author: ASF GitHub Bot Created on: 29/Jan/19 18:41 Start Date: 29/Jan/19 18:41 Worklog Time Spent: 10m Work Description: Ardagan commented on issue #7634: [BEAM-6431] Move state sampler files to runners-core so they can be used in the Java SDK as well URL: https://github.com/apache/beam/pull/7634#issuecomment-458657225 @swegner (committer) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 191854) Time Spent: 1h 40m (was: 1.5h) > Add ExecutionTime metrics to the Beam Java SDK > -- > > Key: BEAM-6431 > URL: https://issues.apache.org/jira/browse/BEAM-6431 > Project: Beam > Issue Type: New Feature > Components: java-fn-execution >Reporter: Alex Amato >Assignee: Alex Amato >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > This will be done by using the Dataflow worker's StateSampler code. I have > put together a refactoring plan > [here|https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#] > This will include estimating the processing time for the start, process and > finish bundle. The python SDK already has an implementation of this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6431) Add ExecutionTime metrics to the Beam Java SDK
[ https://issues.apache.org/jira/browse/BEAM-6431?focusedWorklogId=191855=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-191855 ] ASF GitHub Bot logged work on BEAM-6431: Author: ASF GitHub Bot Created on: 29/Jan/19 18:42 Start Date: 29/Jan/19 18:42 Worklog Time Spent: 10m Work Description: swegner commented on pull request #7634: [BEAM-6431] Move state sampler files to runners-core so they can be used in the Java SDK as well URL: https://github.com/apache/beam/pull/7634 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 191855) Time Spent: 1h 50m (was: 1h 40m) > Add ExecutionTime metrics to the Beam Java SDK > -- > > Key: BEAM-6431 > URL: https://issues.apache.org/jira/browse/BEAM-6431 > Project: Beam > Issue Type: New Feature > Components: java-fn-execution >Reporter: Alex Amato >Assignee: Alex Amato >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > This will be done by using the Dataflow worker's StateSampler code. I have > put together a refactoring plan > [here|https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#] > This will include estimating the processing time for the start, process and > finish bundle. The python SDK already has an implementation of this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6431) Add ExecutionTime metrics to the Beam Java SDK
[ https://issues.apache.org/jira/browse/BEAM-6431?focusedWorklogId=191848=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-191848 ] ASF GitHub Bot logged work on BEAM-6431: Author: ASF GitHub Bot Created on: 29/Jan/19 18:28 Start Date: 29/Jan/19 18:28 Worklog Time Spent: 10m Work Description: ajamato commented on issue #7634: [BEAM-6431] Move state sampler files to runners-core so they can be used in the Java SDK as well URL: https://github.com/apache/beam/pull/7634#issuecomment-458652759 @Ardagan This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 191848) Time Spent: 1.5h (was: 1h 20m) > Add ExecutionTime metrics to the Beam Java SDK > -- > > Key: BEAM-6431 > URL: https://issues.apache.org/jira/browse/BEAM-6431 > Project: Beam > Issue Type: New Feature > Components: java-fn-execution >Reporter: Alex Amato >Assignee: Alex Amato >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > This will be done by using the Dataflow worker's StateSampler code. I have > put together a refactoring plan > [here|https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#] > This will include estimating the processing time for the start, process and > finish bundle. The python SDK already has an implementation of this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6431) Add ExecutionTime metrics to the Beam Java SDK
[ https://issues.apache.org/jira/browse/BEAM-6431?focusedWorklogId=190308=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-190308 ] ASF GitHub Bot logged work on BEAM-6431: Author: ASF GitHub Bot Created on: 26/Jan/19 02:52 Start Date: 26/Jan/19 02:52 Worklog Time Spent: 10m Work Description: ajamato commented on pull request #7634: [BEAM-6431] State sampler move files URL: https://github.com/apache/beam/pull/7634 [BEAM-6431] State sampler move files Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | --- | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 190308) Time Spent: 1h 20m (was: 1h 10m) > Add ExecutionTime metrics to the Beam Java SDK > -- > > Key: BEAM-6431 > URL: https://issues.apache.org/jira/browse/BEAM-6431 > Project: Beam > Issue Type: New Feature > Components:
[jira] [Work logged] (BEAM-6431) Add ExecutionTime metrics to the Beam Java SDK
[ https://issues.apache.org/jira/browse/BEAM-6431?focusedWorklogId=189829=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-189829 ] ASF GitHub Bot logged work on BEAM-6431: Author: ASF GitHub Bot Created on: 25/Jan/19 02:22 Start Date: 25/Jan/19 02:22 Worklog Time Spent: 10m Work Description: swegner commented on pull request #7507: [BEAM-6431] Refactor, Remove references to Dataflow classes in base State Sampling URL: https://github.com/apache/beam/pull/7507 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 189829) Time Spent: 1h 10m (was: 1h) > Add ExecutionTime metrics to the Beam Java SDK > -- > > Key: BEAM-6431 > URL: https://issues.apache.org/jira/browse/BEAM-6431 > Project: Beam > Issue Type: New Feature > Components: java-fn-execution >Reporter: Alex Amato >Assignee: Alex Amato >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > This will be done by using the Dataflow worker's StateSampler code. I have > put together a refactoring plan > [here|https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#] > This will include estimating the processing time for the start, process and > finish bundle. The python SDK already has an implementation of this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6431) Add ExecutionTime metrics to the Beam Java SDK
[ https://issues.apache.org/jira/browse/BEAM-6431?focusedWorklogId=189822=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-189822 ] ASF GitHub Bot logged work on BEAM-6431: Author: ASF GitHub Bot Created on: 25/Jan/19 02:03 Start Date: 25/Jan/19 02:03 Worklog Time Spent: 10m Work Description: ajamato commented on issue #7507: [BEAM-6431] Refactor, Remove references to Dataflow classes in base State Sampling URL: https://github.com/apache/beam/pull/7507#issuecomment-457428834 I have finished all the extra testing which I wanted to perform. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 189822) Time Spent: 1h (was: 50m) > Add ExecutionTime metrics to the Beam Java SDK > -- > > Key: BEAM-6431 > URL: https://issues.apache.org/jira/browse/BEAM-6431 > Project: Beam > Issue Type: New Feature > Components: java-fn-execution >Reporter: Alex Amato >Assignee: Alex Amato >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > This will be done by using the Dataflow worker's StateSampler code. I have > put together a refactoring plan > [here|https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#] > This will include estimating the processing time for the start, process and > finish bundle. The python SDK already has an implementation of this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6431) Add ExecutionTime metrics to the Beam Java SDK
[ https://issues.apache.org/jira/browse/BEAM-6431?focusedWorklogId=189686=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-189686 ] ASF GitHub Bot logged work on BEAM-6431: Author: ASF GitHub Bot Created on: 24/Jan/19 21:03 Start Date: 24/Jan/19 21:03 Worklog Time Spent: 10m Work Description: ajamato commented on issue #7507: [BEAM-6431] [Do not merge] Refactor, Remove references to Dataflow classes in base State Sampling URL: https://github.com/apache/beam/pull/7507#issuecomment-457355342 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 189686) Time Spent: 50m (was: 40m) > Add ExecutionTime metrics to the Beam Java SDK > -- > > Key: BEAM-6431 > URL: https://issues.apache.org/jira/browse/BEAM-6431 > Project: Beam > Issue Type: New Feature > Components: java-fn-execution >Reporter: Alex Amato >Assignee: Alex Amato >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > This will be done by using the Dataflow worker's StateSampler code. I have > put together a refactoring plan > [here|https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#] > This will include estimating the processing time for the start, process and > finish bundle. The python SDK already has an implementation of this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6431) Add ExecutionTime metrics to the Beam Java SDK
[ https://issues.apache.org/jira/browse/BEAM-6431?focusedWorklogId=189244=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-189244 ] ASF GitHub Bot logged work on BEAM-6431: Author: ASF GitHub Bot Created on: 24/Jan/19 00:49 Start Date: 24/Jan/19 00:49 Worklog Time Spent: 10m Work Description: ajamato commented on issue #7507: [BEAM-6431] [Do not merge] Refactor, Remove references to Dataflow classes in base State Sampling URL: https://github.com/apache/beam/pull/7507#issuecomment-457023765 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 189244) Time Spent: 40m (was: 0.5h) > Add ExecutionTime metrics to the Beam Java SDK > -- > > Key: BEAM-6431 > URL: https://issues.apache.org/jira/browse/BEAM-6431 > Project: Beam > Issue Type: New Feature > Components: java-fn-execution >Reporter: Alex Amato >Assignee: Alex Amato >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > This will be done by using the Dataflow worker's StateSampler code. I have > put together a refactoring plan > [here|https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#] > This will include estimating the processing time for the start, process and > finish bundle. The python SDK already has an implementation of this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6431) Add ExecutionTime metrics to the Beam Java SDK
[ https://issues.apache.org/jira/browse/BEAM-6431?focusedWorklogId=186644=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-186644 ] ASF GitHub Bot logged work on BEAM-6431: Author: ASF GitHub Bot Created on: 18/Jan/19 01:01 Start Date: 18/Jan/19 01:01 Worklog Time Spent: 10m Work Description: pabloem commented on issue #7507: [BEAM-6431] Refactor, Remove references to Dataflow classes in base State Sampling URL: https://github.com/apache/beam/pull/7507#issuecomment-455389115 Left comments in design doc. LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 186644) Time Spent: 0.5h (was: 20m) > Add ExecutionTime metrics to the Beam Java SDK > -- > > Key: BEAM-6431 > URL: https://issues.apache.org/jira/browse/BEAM-6431 > Project: Beam > Issue Type: New Feature > Components: java-fn-execution >Reporter: Alex Amato >Assignee: Alex Amato >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > This will be done by using the Dataflow worker's StateSampler code. I have > put together a refactoring plan > [here|https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#] > This will include estimating the processing time for the start, process and > finish bundle. The python SDK already has an implementation of this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6431) Add ExecutionTime metrics to the Beam Java SDK
[ https://issues.apache.org/jira/browse/BEAM-6431?focusedWorklogId=186089=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-186089 ] ASF GitHub Bot logged work on BEAM-6431: Author: ASF GitHub Bot Created on: 17/Jan/19 00:47 Start Date: 17/Jan/19 00:47 Worklog Time Spent: 10m Work Description: ajamato commented on issue #7507: [BEAM-6431] Refactor, Remove references to Dataflow classes in base State Sampling URL: https://github.com/apache/beam/pull/7507#issuecomment-454999807 @Ardagan This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 186089) Time Spent: 20m (was: 10m) > Add ExecutionTime metrics to the Beam Java SDK > -- > > Key: BEAM-6431 > URL: https://issues.apache.org/jira/browse/BEAM-6431 > Project: Beam > Issue Type: New Feature > Components: java-fn-execution >Reporter: Alex Amato >Assignee: Alex Amato >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > This will be done by using the Dataflow worker's StateSampler code. I have > put together a refactoring plan > [here|https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#] > This will include estimating the processing time for the start, process and > finish bundle. The python SDK already has an implementation of this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6431) Add ExecutionTime metrics to the Beam Java SDK
[ https://issues.apache.org/jira/browse/BEAM-6431?focusedWorklogId=186088=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-186088 ] ASF GitHub Bot logged work on BEAM-6431: Author: ASF GitHub Bot Created on: 17/Jan/19 00:47 Start Date: 17/Jan/19 00:47 Worklog Time Spent: 10m Work Description: ajamato commented on issue #7507: [BEAM-6431] Refactor, Remove references to Dataflow classes in base State Sampling URL: https://github.com/apache/beam/pull/7507#issuecomment-454999774 @swegner @pabloem This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 186088) Time Spent: 10m Remaining Estimate: 0h > Add ExecutionTime metrics to the Beam Java SDK > -- > > Key: BEAM-6431 > URL: https://issues.apache.org/jira/browse/BEAM-6431 > Project: Beam > Issue Type: New Feature > Components: java-fn-execution >Reporter: Alex Amato >Assignee: Alex Amato >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > This will be done by using the Dataflow worker's StateSampler code. I have > put together a refactoring plan > [here|https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#] > This will include estimating the processing time for the start, process and > finish bundle. The python SDK already has an implementation of this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
