[
https://issues.apache.org/jira/browse/BEAM-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16275261#comment-16275261
]
ASF GitHub Bot commented on BEAM-1630:
--------------------------------------
chamikaramj commented on a change in pull request #4064: [BEAM-1630] Adds
support for processing Splittable DoFns using DirectRunner.
URL: https://github.com/apache/beam/pull/4064#discussion_r153386073
##########
File path: sdks/python/apache_beam/runners/direct/direct_runner.py
##########
@@ -86,19 +86,25 @@ def from_runner_api_parameter(payload, context):
context.windowing_strategies.get_by_id(payload.value))
-class DirectRunner(PipelineRunner):
- """Executes a single pipeline on the local machine."""
-
+def _get_transform_overrides():
# A list of PTransformOverride objects to be applied before running a
pipeline
# using DirectRunner.
# Currently this only works for overrides where the input and output types do
# not change.
- # For internal SDK use only. This should not be updated by Beam pipeline
- # authors.
- _PTRANSFORM_OVERRIDES = []
+ # For internal SDK use only.
+
+ from apache_beam.runners.direct.sdf_direct_runner import
ProcessKeyedElementsViaKeyedWorkItemsOverride
+ from apache_beam.runners.sdf_common import SplittableParDoOverride
Review comment:
Yeah, importing this at top results in a circular dependency and this was
the main reason for this change. Added a comment. In addition to that, making
this a method allows us to add any additional logic here if needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Splittable DoFn to Python SDK
> ---------------------------------
>
> Key: BEAM-1630
> URL: https://issues.apache.org/jira/browse/BEAM-1630
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
> Reporter: Chamikara Jayalath
> Assignee: Chamikara Jayalath
>
> Splittable DoFn [1] is currently being implemented for Java SDK [2]. We
> should add this to Python SDK as well.
> Following document proposes an API for this.
> https://docs.google.com/document/d/1h_zprJrOilivK2xfvl4L42vaX4DMYGfH1YDmi-s_ozM/edit?usp=sharing
> [1] https://s.apache.org/splittable-do-fn
> [2]
> https://lists.apache.org/thread.html/0ce61ac162460a149d5c93cdface37cc383f8030fe86ca09e5699b18@%3Cdev.beam.apache.org%3E
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
