Re: BEAM-831 ParDo chaining for Apex runner

Hi All, I did the implementation as you mentioned and created a PR ( https://github.com/apache/beam/pull/2216) for CI and also for code related discussion. I'm writing unit tests for the code changes. If you get time, please have a look at the code and provide the feedback. Thanks in advance,

Re: BEAM-831 ParDo chaining for Apex runner

Thomas, Thanks. I'll work on optimization as suggested by you. -Chinmay. On Mon, Feb 27, 2017 at 4:16 AM, Thomas Weise wrote: > Chinmay, > > I would recommend to collapse ParDo sequences to the maximum extend > possible using THREAD_LOCAL affinity. The Apex runner has a

Re: BEAM-831 ParDo chaining for Apex runner

Chinmay, I would recommend to collapse ParDo sequences to the maximum extend possible using THREAD_LOCAL affinity. The Apex runner has a configuration file option that can be used to tune the chaining when needed ( https://issues.apache.org/jira/browse/BEAM-980). What you build into the pipeline

BEAM-831 ParDo chaining for Apex runner

Dear Community, I'm working on BEAM-831 to implement ParDo chaining for Apache Apex Runner. As suggested on Jira, chaining needs to be done using Stream locality of Apache Apex engine. I got some links from Eugene Kirpichov on the Jira. I'm currently focusing on producer-consumer fusion