[
https://issues.apache.org/jira/browse/BEAM-9167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Burke reassigned BEAM-9167:
----------------------------------
Assignee: Robert Burke
> Reduce overhead of Go SDK side metrics
> --------------------------------------
>
> Key: BEAM-9167
> URL: https://issues.apache.org/jira/browse/BEAM-9167
> Project: Beam
> Issue Type: Improvement
> Components: sdk-go
> Reporter: Robert Burke
> Assignee: Robert Burke
> Priority: Major
>
> Locking overhead due to the global store and local caches of SDK counter data
> can dominate certain workloads, which means we can do better.
> Instead of having a global store of metrics data to extract counters, we
> should use per ptransform (or per bundle) counter sets, which would avoid
> requiring locking per counter operation. The main detriment compared to the
> current implementation is that a user would need to add their own locking if
> they were to spawn multiple goroutines to process a Bundle's work in a DoFn.
> Given that self multithreaded DoFns aren't recommended/safe in Java, largely
> impossible in Python, and the other beam Go SDK provided constructs (like
> Iterators and Emitters) are not thread safe, this is a small concern,
> provided the documentation is clear on this.
> Removing the locking and switching to atomic ops reduces the overhead
> significantly in example jobs and in the benchmarks.
> Related: https://issues.apache.org/jira/browse/BEAM-6541
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
