[ https://issues.apache.org/jira/browse/BEAM-9167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Burke reassigned BEAM-9167: ---------------------------------- Assignee: Robert Burke > Reduce overhead of Go SDK side metrics > -------------------------------------- > > Key: BEAM-9167 > URL: https://issues.apache.org/jira/browse/BEAM-9167 > Project: Beam > Issue Type: Improvement > Components: sdk-go > Reporter: Robert Burke > Assignee: Robert Burke > Priority: Major > > Locking overhead due to the global store and local caches of SDK counter data > can dominate certain workloads, which means we can do better. > Instead of having a global store of metrics data to extract counters, we > should use per ptransform (or per bundle) counter sets, which would avoid > requiring locking per counter operation. The main detriment compared to the > current implementation is that a user would need to add their own locking if > they were to spawn multiple goroutines to process a Bundle's work in a DoFn. > Given that self multithreaded DoFns aren't recommended/safe in Java, largely > impossible in Python, and the other beam Go SDK provided constructs (like > Iterators and Emitters) are not thread safe, this is a small concern, > provided the documentation is clear on this. > Removing the locking and switching to atomic ops reduces the overhead > significantly in example jobs and in the benchmarks. > Related: https://issues.apache.org/jira/browse/BEAM-6541 -- This message was sent by Atlassian Jira (v8.3.4#803005)