pabloem commented on a change in pull request #12713:
URL: https://github.com/apache/beam/pull/12713#discussion_r482224353
##########
File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
##########
@@ -118,13 +118,15 @@ class DataflowRunner(PipelineRunner):
from apache_beam.runners.dataflow.ptransform_overrides import
CreatePTransformOverride
from apache_beam.runners.dataflow.ptransform_overrides import
JrhReadPTransformOverride
from apache_beam.runners.dataflow.ptransform_overrides import
ReadPTransformOverride
+ from apache_beam.runners.dataflow.ptransform_overrides import
ReadBigQuerySourcePTransformOverride
from apache_beam.runners.dataflow.ptransform_overrides import
NativeReadPTransformOverride
# These overrides should be applied before the proto representation of the
# graph is created.
_PTRANSFORM_OVERRIDES = [
CombineValuesPTransformOverride(),
NativeReadPTransformOverride(),
+ ReadBigQuerySourcePTransformOverride(),
Review comment:
.. an issue with this, though, is that BigQuerySource is wrapped by a
Read transform. So we'd have `Read(BigQuerySource(...))` - and so we would have
then a `Read(ReadFromBigQuery(...))`, which would not work. We can also return
the `_CustomBigQuerySource`, but it would be missing the clean up after the
export jobs. Thoughts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
