[
https://issues.apache.org/jira/browse/BEAM-8418?focusedWorklogId=332111&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-332111
]
ASF GitHub Bot logged work on BEAM-8418:
----------------------------------------
Author: ASF GitHub Bot
Created on: 22/Oct/19 17:31
Start Date: 22/Oct/19 17:31
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #9822: [BEAM-8418]
Use base64 string for representing impulse payload in DF runner legacy codepath.
URL: https://github.com/apache/beam/pull/9822
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 332111)
Time Spent: 2h 50m (was: 2h 40m)
> Fix handling of Impulse transform in Dataflow runner.
> ------------------------------------------------------
>
> Key: BEAM-8418
> URL: https://issues.apache.org/jira/browse/BEAM-8418
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Valentyn Tymofieiev
> Assignee: Robert Bradshaw
> Priority: Major
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Following pipeline fails on Dataflow runner unless we use beam_fn_api
> experiment.
> {noformat}
> class NoOpDoFn(beam.DoFn):
> def process(self, element):
> return element
> p = beam.Pipeline(options=pipeline_options)
> _ = p | beam.Impulse() | beam.ParDo(NoOpDoFn())
> result = p.run()
> {noformat}
> The reason is that we encode Impluse payload using url-escaping in [1], while
> Dataflow runner expects base64 encoding in non-fnapi mode. In FnApi mode, DF
> runner expects URL escaping.
> We should fix or reconcile the encoding in non-FnAPI path, and add a
> ValidatesRunner test that catches this error.
> [1]
> https://github.com/apache/beam/blob/12d07745835e1b9c1e824b83beeeadf63ab4b234/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L633
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
