pavansai1997 opened a new pull request, #77153:
URL: https://github.com/apache/airflow/pull/77153
`SqlToS3Operator` converts a templated `page_size` to `int` in a
`render_template_fields` override (#75730). A mapped task never calls that
method: the task runner calls `MappedOperator.render_template_fields`, which
renders through `_do_render_template_fields` on the unmapped task. So in a
mapped task `page_size` stayed a string, and the task uploaded the first page
and then failed on `len(df) < self.page_size` with `TypeError: '<' not
supported between instances of 'int' and 'str'`. With `page_size` rendering
`"0"`, a mapped task paged instead of writing one file.
This moves the conversion into `_do_render_template_fields`, which plain and
mapped tasks both go through. It runs after the fields are rendered and only
for the operator itself (`parent is self`), not for nested templated objects.
`common.ai`'s `AgentOperator` uses the same hook for the same reason. Plain
tasks are unchanged: `BaseOperator.render_template_fields` calls
`_do_render_template_fields` with the operator as `parent`. Airflow 2.11's
`BaseOperator` and `MappedOperator` call `_do_render_template_fields` with the
same signature, so the fix applies there too.
Running a plain and a mapped task with `page_size="{{ ps }}"` through
`render_template_fields` and `execute()`, with a SQL hook that returns pages of
2, 2 and 1 rows and a mocked `S3Hook`:
| `ps` | task | before | after |
|---|---|---|---|
| `2` | plain | 3 uploads | 3 uploads |
| `2` | mapped | 1 upload, then `TypeError` | 3 uploads |
| `0` | plain | 1 unpaged upload | 1 unpaged upload |
| `0` | mapped | 1 paged upload (`LIMIT 0`), then `TypeError` | 1 unpaged
upload |
The new test renders a `.partial().expand()` task with `create_runtime_ti`
and `ti.render_templates()`, the method the task runner calls, and fails
without this change. It is skipped before Airflow 3.0 because the fixture needs
the Task SDK.
`GenericTransfer` in `common.sql` converts `page_size` in a
`render_template_fields` override in the same way. It is a separate
distribution and I have not reproduced the problem there, so it is not changed
here.
closes: #77151
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Code Puppy
Generated-by: Code Puppy following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]