y1chi commented on code in PR #17323:
URL: https://github.com/apache/beam/pull/17323#discussion_r854578401
##########
sdks/python/apache_beam/runners/dataflow/dataflow_runner.py:
##########
@@ -393,6 +393,72 @@ def _adjust_pipeline_for_dataflow_v2(self, pipeline):
def _check_for_unsupported_features_on_non_portable_worker(self, pipeline):
pipeline.visit(self.combinefn_visitor())
+ def _enable_sdk_prebuild_if_applicable(self, options):
+ setup_options = options.view_as(SetupOptions)
+ # SDK container image pre-build explicitly enabled or disabled.
+ if (setup_options.prebuild_sdk_container_engine or
+ setup_options.disable_sdk_container_prebuild):
+ return
+ from apache_beam.runners.dataflow.internal import apiclient
+ # prebuild_sdk_container_engine only works with fnapi jobs.
+ if (not apiclient._use_fnapi(options) and
Review Comment:
On a side note, I guess the prebuilding doesn't really involve beam_fn_api,
so I'm wondering if we can backport the boot.go change to non-fnapi python
worker containers, I think the prebuilding has value for jobs on legacy python
batch worker as well, WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
