[jira] [Commented] (BEAM-9934) Resolve differences in beam:metric:element_count:v1 implementations
[ https://issues.apache.org/jira/browse/BEAM-9934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105866#comment-17105866 ] Luke Cwik commented on BEAM-9934: - 2.22 > Resolve differences in beam:metric:element_count:v1 implementations > --- > > Key: BEAM-9934 > URL: https://issues.apache.org/jira/browse/BEAM-9934 > Project: Beam > Issue Type: Bug > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Major > Fix For: 2.22.0 > > > The [element > count|https://github.com/apache/beam/blob/d82d061aa303430f3d2853f397f3130fae6200cd/model/pipeline/src/main/proto/metrics.proto#L206] > metric represents the number of elements within a PCollection and is > interpreted differently across the Beam SDK versions. > In the [Java > SDK|https://github.com/apache/beam/blob/d82d061aa303430f3d2853f397f3130fae6200cd/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/PCollectionConsumerRegistry.java#L207] > this represents the number of elements and includes how many windows those > elements are in. This metric is incremented as soon as the element has been > output. > In the [Python > SDK|https://github.com/apache/beam/blame/bfd151aa4c3aad29f3aea6482212ff8543ded8d7/sdks/python/apache_beam/runners/worker/opcounters.py#L247] > this represents the number of elements and doesn't include how many windows > those elements are in. The metric is also only incremented after the element > has finished processing. > The [Go > SDK|https://github.com/apache/beam/blob/7097850daa46674b88425a124bc442fc8ce0dcb8/sdks/go/pkg/beam/core/runtime/exec/datasource.go#L260] > does the same thing as Python. > Traditionally in Dataflow this has always been the exploded window element > count and the counter is incremented as soon as the element is output. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9934) Resolve differences in beam:metric:element_count:v1 implementations
[ https://issues.apache.org/jira/browse/BEAM-9934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105846#comment-17105846 ] Kyle Weaver commented on BEAM-9934: --- What's the status on this one? Are we going to try to fix this for 2.21.0, or wait until 2.22.0? > Resolve differences in beam:metric:element_count:v1 implementations > --- > > Key: BEAM-9934 > URL: https://issues.apache.org/jira/browse/BEAM-9934 > Project: Beam > Issue Type: Bug > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Major > Fix For: 2.21.0 > > > The [element > count|https://github.com/apache/beam/blob/d82d061aa303430f3d2853f397f3130fae6200cd/model/pipeline/src/main/proto/metrics.proto#L206] > metric represents the number of elements within a PCollection and is > interpreted differently across the Beam SDK versions. > In the [Java > SDK|https://github.com/apache/beam/blob/d82d061aa303430f3d2853f397f3130fae6200cd/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/PCollectionConsumerRegistry.java#L207] > this represents the number of elements and includes how many windows those > elements are in. This metric is incremented as soon as the element has been > output. > In the [Python > SDK|https://github.com/apache/beam/blame/bfd151aa4c3aad29f3aea6482212ff8543ded8d7/sdks/python/apache_beam/runners/worker/opcounters.py#L247] > this represents the number of elements and doesn't include how many windows > those elements are in. The metric is also only incremented after the element > has finished processing. > The [Go > SDK|https://github.com/apache/beam/blob/7097850daa46674b88425a124bc442fc8ce0dcb8/sdks/go/pkg/beam/core/runtime/exec/datasource.go#L260] > does the same thing as Python. > Traditionally in Dataflow this has always been the exploded window element > count and the counter is incremented as soon as the element is output. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9934) Resolve differences in beam:metric:element_count:v1 implementations
[ https://issues.apache.org/jira/browse/BEAM-9934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103193#comment-17103193 ] Robert Bradshaw commented on BEAM-9934: --- This inconsistency should be fixed, but has been with us ever since Java and Python both reported element counts, so downgrading from a blocker. I did, however, file https://issues.apache.org/jira/browse/BEAM-9945 which is a blocker. > Resolve differences in beam:metric:element_count:v1 implementations > --- > > Key: BEAM-9934 > URL: https://issues.apache.org/jira/browse/BEAM-9934 > Project: Beam > Issue Type: Bug > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Blocker > Fix For: 2.21.0 > > > The [element > count|https://github.com/apache/beam/blob/d82d061aa303430f3d2853f397f3130fae6200cd/model/pipeline/src/main/proto/metrics.proto#L206] > metric represents the number of elements within a PCollection and is > interpreted differently across the Beam SDK versions. > In the [Java > SDK|https://github.com/apache/beam/blob/d82d061aa303430f3d2853f397f3130fae6200cd/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/PCollectionConsumerRegistry.java#L207] > this represents the number of elements and includes how many windows those > elements are in. This metric is incremented as soon as the element has been > output. > In the [Python > SDK|https://github.com/apache/beam/blame/bfd151aa4c3aad29f3aea6482212ff8543ded8d7/sdks/python/apache_beam/runners/worker/opcounters.py#L247] > this represents the number of elements and doesn't include how many windows > those elements are in. The metric is also only incremented after the element > has finished processing. > The [Go > SDK|https://github.com/apache/beam/blob/7097850daa46674b88425a124bc442fc8ce0dcb8/sdks/go/pkg/beam/core/runtime/exec/datasource.go#L260] > does the same thing as Python. > Traditionally in Dataflow this has always been the exploded window element > count and the counter is incremented as soon as the element is output. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9934) Resolve differences in beam:metric:element_count:v1 implementations
[ https://issues.apache.org/jira/browse/BEAM-9934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102956#comment-17102956 ] Luke Cwik commented on BEAM-9934: - Marked as blocker for now if decide that this should get fixed in Python. > Resolve differences in beam:metric:element_count:v1 implementations > --- > > Key: BEAM-9934 > URL: https://issues.apache.org/jira/browse/BEAM-9934 > Project: Beam > Issue Type: Bug > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Major > Fix For: 2.21.0 > > > The [element > count|https://github.com/apache/beam/blob/d82d061aa303430f3d2853f397f3130fae6200cd/model/pipeline/src/main/proto/metrics.proto#L206] > metric represents the number of elements within a PCollection and is > interpreted differently across the Beam SDK versions. > In the [Java > SDK|https://github.com/apache/beam/blob/d82d061aa303430f3d2853f397f3130fae6200cd/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/PCollectionConsumerRegistry.java#L207] > this represents the number of elements and includes how many windows those > elements are in. This metric is incremented as soon as the element has been > output. > In the [Python > SDK|https://github.com/apache/beam/blame/bfd151aa4c3aad29f3aea6482212ff8543ded8d7/sdks/python/apache_beam/runners/worker/opcounters.py#L247] > this represents the number of elements and doesn't include how many windows > those elements are in. The metric is also only incremented after the element > has finished processing. > The [Go > SDK|https://github.com/apache/beam/blob/7097850daa46674b88425a124bc442fc8ce0dcb8/sdks/go/pkg/beam/core/runtime/exec/datasource.go#L260] > does the same thing as Python. > Traditionally in Dataflow this has always been the exploded window element > count and the counter is incremented as soon as the element is output. -- This message was sent by Atlassian Jira (v8.3.4#803005)
