jonathangosling opened a new pull request, #71675:
URL: https://github.com/apache/airflow/pull/71675
`ignore_unknown_values=True` silently does nothing under the operator's
default `autodetect=True`. The load job's schema is inferred from the same
source data the values are checked against, so no source field can be unknown
to it. Users set the flag expecting fields absent from the destination table to
be dropped, and instead get a schema mismatch with nothing in the logs
indicating the flag was ignored.
This warns when the assembled load configuration has `autodetect` truthy,
`ignoreUnknownValues` set, and no `schema`, pointing at `autodetect=None` - the
setting that makes BigQuery use the destination table's own schema.
The check reads the assembled `configuration["load"]` rather than the
operator attributes, because `src_fmt_configs` and `extra_config` can override
any of those three keys after they are set (all three are in `valid_configs`
for the relevant source formats). Two of the added test cases cover exactly
that.
It also fixes the `autodetect` docstring, which actively pointed the wrong
way: it said the parameter "must be set to True if 'schema_fields' and
'schema_object' are undefined" (`None` is also valid, and is the documented way
to load into an existing table), and that it is "suggested to set to True if
table are create outside of Airflow" (for a table created outside Airflow,
`True` is precisely what triggers the mismatch - `None` is what works). All
three states are now documented, and `autodetect` gets a `bool | None`
annotation matching the existing `is False` check at the schema-object guard.
Deliberately out of scope: changing the default, making `autodetect=False`
fall back to the table schema, and redesigning the flag. The three-state
parameter is an artifact of `"autodetect": self.autodetect` being serialized
unconditionally, so `None` is the only way to express "omit the field" - worth
revisiting separately, but not in a bugfix PR.
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull
request.
Write a good git commit message following this guide:
https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the
documentation.
For user-facing UI changes, please attach before/after screenshots (or a
short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for
a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we
sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Opus 5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
