jonathangosling commented on code in PR #71675:
URL: https://github.com/apache/airflow/pull/71675#discussion_r3792563518
##########
providers/google/src/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py:
##########
@@ -162,12 +164,21 @@ class GCSToBigQueryOperator(BaseOperator):
by one or more columns. BigQuery supports clustering for both
partitioned and
non-partitioned tables. The order of columns given determines the sort
order.
Not applicable for external tables.
- :param autodetect: [Optional] Indicates if we should automatically infer
the
- options and schema for CSV and JSON sources. (Default: ``True``).
- Parameter must be set to True if 'schema_fields' and 'schema_object'
are undefined.
- It is suggested to set to True if table are create outside of Airflow.
- If autodetect is None and no schema is provided (neither via
schema_fields
- nor a schema_object), assume the table already exists.
+ :param autodetect: [Optional] Indicates whether the options and schema
should be inferred
+ from the source data for CSV and JSON sources. (Default: ``True``).
This parameter is
+ three-state:
+
+ * ``True`` - infer the schema from the source files. The inferred
schema is sent to
+ BigQuery as the load job's schema, so ``ignore_unknown_values`` has
no effect: no field
+ in the source data can be unknown to a schema derived from that same
data. If the
+ destination table already exists and the inferred schema differs
from it, the load
+ fails with a schema mismatch unless ``schema_update_options``
permits the change.
+ * ``False`` - do not infer the schema. One of ``schema_fields`` or
``schema_object``
+ must be provided, otherwise an exception is raised.
+ * ``None`` - neither infer a schema nor supply one, so the destination
table's own
+ schema is used. The table must already exist. This is the setting to
use when you want
+ ``ignore_unknown_values`` to drop fields that are present in the
source data but absent
+ from the destination table.
Review Comment:
Good point, I've updated the docstring with your suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
