jonathangosling commented on code in PR #71675:
URL: https://github.com/apache/airflow/pull/71675#discussion_r3792564344
##########
providers/google/src/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py:
##########
@@ -638,6 +649,23 @@ def _create_external_table(self):
self.log.info("External table created successfully: %s",
self.destination_project_dataset_table)
return table_obj_api_repr
+ def _warn_if_ignore_unknown_values_is_ineffective(self) -> None:
+ # Checked against the assembled load config rather than the operator
attributes, because
+ # src_fmt_configs and extra_config can still override any of these
three keys.
+ load_config = self.configuration["load"]
+ if (
+ load_config.get("autodetect")
+ and load_config.get("ignoreUnknownValues")
+ and "schema" not in load_config
+ ):
+ self.log.warning(
+ "`ignore_unknown_values` is set but will have no effect. With
`autodetect` enabled "
+ "and no schema supplied, the load job's schema is inferred
from the source data "
+ "itself, so no source field can be unknown to it. To drop
source fields that are "
+ "absent from an existing destination table, set
`autodetect=None` so that the "
+ "destination table's own schema is used instead."
+ )
Review Comment:
Updated:
- Inlined the check and warning.
- Made the message less verbose. I've used your suggestion, which covers the
problem and the suggested fix more concisely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
