pavansai1997 opened a new issue, #77151:
URL: https://github.com/apache/airflow/issues/77151
### Under which category would you file this issue?
Providers
### Apache Airflow version
main (development)
### What happened and how to reproduce it?
With `page_size` templated, `SqlToS3Operator` (#75730, released in
`apache-airflow-providers-amazon` 9.40.0) works in a plain task but crashes in
a mapped task after uploading the first page.
`SqlToS3Operator.render_template_fields` converts the rendered `page_size`
to `int`
(`providers/amazon/src/airflow/providers/amazon/aws/transfers/sql_to_s3.py`,
around line 280). For a mapped task, the task runner calls
`MappedOperator.render_template_fields`, which unmaps the task and calls
`unmapped_task._do_render_template_fields(...)` directly
(`task-sdk/src/airflow/sdk/definitions/mappedoperator.py`), so the operator's
override never runs. `page_size` stays the string `"2"`: it is truthy, so
`execute()` starts paging, and the first comparison `len(df) < self.page_size`
raises.
```python
with DAG("sql_to_s3_page_size", schedule=None,
start_date=datetime.datetime(2026, 1, 1)):
common = {"query": "SELECT id FROM t", "s3_bucket": "bucket",
"sql_conn_id": "db", "page_size": "{{ 1 + 1 }}"}
plain = SqlToS3Operator(task_id="plain", s3_key="plain", **common)
mapped = SqlToS3Operator.partial(task_id="mapped",
**common).expand(s3_key=["mapped"])
```
Rendering each task the way the task runner does
(`task.render_template_fields(context)`) and running `execute()` with a SQL
hook that returns pages of 2, 2 and 1 rows and a mocked `S3Hook`:
```
plain page_size=2 queries=3 uploads=['plain_0.csv', 'plain_1.csv',
'plain_2.csv'] -> OK
mapped page_size='2' queries=1 uploads=['mapped_0.csv'] -> TypeError: '<'
not supported between instances of 'int' and 'str'
```
The mapped task has already uploaded `mapped_0.csv` when it fails, and each
retry repeats that. Any templated value behaves the same way, for example one
pulled from XCom, because templates render to strings unless the Dag uses
native rendering. With `page_size` rendering `"0"`, the plain task does not
page (one `plain.csv` upload), but the mapped task pages, uploads
`mapped_0.csv` and fails with the same `TypeError`. The negative-value check is
skipped for mapped tasks as well.
None of the tests in `test_sql_to_s3.py` use `.partial()` or `.expand()`, so
they do not cover this.
### What you think should happen instead?
A mapped task should get the same integer `page_size` as a plain one. Doing
the conversion in `_do_render_template_fields`, which
`MappedOperator.render_template_fields` calls on the unmapped task (its comment
describes overriding it for this purpose), or in `execute()`, would cover both
paths.
`GenericTransfer`
(`providers/common/sql/src/airflow/providers/common/sql/operators/generic_transfer.py`)
converts `page_size` in a `render_template_fields` override in the same way,
so it is likely affected too. I have not reproduced that one.
### Operating System
macOS
### Deployment
Virtualenv installation
### Apache Airflow Provider(s)
amazon
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon 9.40.0 (reproduced against `main` at
5e2e1417f2; `sql_to_s3.py` is unchanged since #75730)
### Anything else?
Introduced by #75730.
cc @o-nikolas (code owner for `providers/amazon/`), @dabla (author of
#75730), @vincbeck (reviewer of #75730)
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]