[
https://issues.apache.org/jira/browse/BEAM-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887964#comment-15887964
]
Amit Sela commented on BEAM-1556:
---------------------------------
My line of thought about this being in the SDK (or better, the Runner API) is
because the runner would have to init. the registration for every instance,
workers mostly (the implementation of {{PipelineRunner}} would probably take
care of it for the "Driver" instance).
Since not all {{DoFn}} require this, and not all readers/writes.. so it's
either init. all the time (regardless if needed or not) or the runner would
have to patch-up for every new use case: Read, Write, DoFn...
I'm not sure I'm going to like the following suggestion (fighting with myself a
bit here), but how about a {{FileSystemContext}} ? and the runner would have to
initialize in it and pass it on to the SDK ?
Not sure here.. thoughts ?
> Spark executors need to register IO factories
> ---------------------------------------------
>
> Key: BEAM-1556
> URL: https://issues.apache.org/jira/browse/BEAM-1556
> Project: Beam
> Issue Type: Bug
> Components: runner-spark
> Reporter: Frances Perry
> Assignee: Jean-Baptiste Onofré
>
> The Spark executors need to call IOChannelUtils.registerIOFactories(options)
> in order to support GCS file and make the default WordCount example work.
> Context in this thread:
> https://lists.apache.org/thread.html/469a139c9eb07e64e514cdea42ab8000678ab743794a090c365205d7@%3Cuser.beam.apache.org%3E
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
