robertwb commented on a change in pull request #12713:
URL: https://github.com/apache/beam/pull/12713#discussion_r481446948
##########
File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py
##########
@@ -259,18 +260,6 @@ def test_bigquery_read_streaming_fail(self):
PipelineOptions(self.default_properties)) as p:
_ = p | beam.io.Read(beam.io.BigQuerySource('some.table'))
- def test_biqquery_read_fn_api_fail(self):
Review comment:
Wouldn't we still want an error most fo the time? (E.g. this is still
broken on OSS runners.)
##########
File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
##########
@@ -118,13 +118,15 @@ class DataflowRunner(PipelineRunner):
from apache_beam.runners.dataflow.ptransform_overrides import
CreatePTransformOverride
from apache_beam.runners.dataflow.ptransform_overrides import
JrhReadPTransformOverride
from apache_beam.runners.dataflow.ptransform_overrides import
ReadPTransformOverride
+ from apache_beam.runners.dataflow.ptransform_overrides import
ReadBigQuerySourcePTransformOverride
from apache_beam.runners.dataflow.ptransform_overrides import
NativeReadPTransformOverride
# These overrides should be applied before the proto representation of the
# graph is created.
_PTRANSFORM_OVERRIDES = [
CombineValuesPTransformOverride(),
NativeReadPTransformOverride(),
+ ReadBigQuerySourcePTransformOverride(),
Review comment:
The problem with putting this as an override here is that it would be
called *after* the beam protos are created.
Instead of implementing this as a (Dataflow-only) override, it'd be better
to change `beam.io.bigquery.BigQuerySource` into a function that returns a
`ReadFromBigquery` transform.
We could either remove the native `BigQuerySource` altogether, or add a
(temporary) keyword argument `use_native_source` to our new `BigQuerySource`
function to branch on as an escape hatch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
