[ https://issues.apache.org/jira/browse/BEAM-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16957267#comment-16957267 ]
Robert Bradshaw commented on BEAM-8418: --------------------------------------- Note that this is a Dataflow-only issue. The FnAPI doesn't have a way to specifying what element should be used for Impulse (and Dataflow should probably be updated accordingly). > Fix handling of Impulse transform in Dataflow runner. > ------------------------------------------------------ > > Key: BEAM-8418 > URL: https://issues.apache.org/jira/browse/BEAM-8418 > Project: Beam > Issue Type: Bug > Components: sdk-py-core > Reporter: Valentyn Tymofieiev > Assignee: Robert Bradshaw > Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > Following pipeline fails on Dataflow runner unless we use beam_fn_api > experiment. > {noformat} > class NoOpDoFn(beam.DoFn): > def process(self, element): > return element > p = beam.Pipeline(options=pipeline_options) > _ = p | beam.Impulse() | beam.ParDo(NoOpDoFn()) > result = p.run() > {noformat} > The reason is that we encode Impluse payload using url-escaping in [1], while > Dataflow runner expects base64 encoding in non-fnapi mode. In FnApi mode, DF > runner expects URL escaping. > We should fix or reconcile the encoding in non-FnAPI path, and add a > ValidatesRunner test that catches this error. > [1] > https://github.com/apache/beam/blob/12d07745835e1b9c1e824b83beeeadf63ab4b234/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L633 -- This message was sent by Atlassian Jira (v8.3.4#803005)