pavansai1997 opened a new pull request, #77153:
URL: https://github.com/apache/airflow/pull/77153

   `SqlToS3Operator` converts a templated `page_size` to `int` in a 
`render_template_fields` override (#75730). A mapped task never calls that 
method: the task runner calls `MappedOperator.render_template_fields`, which 
renders through `_do_render_template_fields` on the unmapped task. So in a 
mapped task `page_size` stayed a string, and the task uploaded the first page 
and then failed on `len(df) < self.page_size` with `TypeError: '<' not 
supported between instances of 'int' and 'str'`. With `page_size` rendering 
`"0"`, a mapped task paged instead of writing one file.
   
   This moves the conversion into `_do_render_template_fields`, which plain and 
mapped tasks both go through. It runs after the fields are rendered and only 
for the operator itself (`parent is self`), not for nested templated objects. 
`common.ai`'s `AgentOperator` uses the same hook for the same reason. Plain 
tasks are unchanged: `BaseOperator.render_template_fields` calls 
`_do_render_template_fields` with the operator as `parent`. Airflow 2.11's 
`BaseOperator` and `MappedOperator` call `_do_render_template_fields` with the 
same signature, so the fix applies there too.
   
   Running a plain and a mapped task with `page_size="{{ ps }}"` through 
`render_template_fields` and `execute()`, with a SQL hook that returns pages of 
2, 2 and 1 rows and a mocked `S3Hook`:
   
   | `ps` | task | before | after |
   |---|---|---|---|
   | `2` | plain | 3 uploads | 3 uploads |
   | `2` | mapped | 1 upload, then `TypeError` | 3 uploads |
   | `0` | plain | 1 unpaged upload | 1 unpaged upload |
   | `0` | mapped | 1 paged upload (`LIMIT 0`), then `TypeError` | 1 unpaged 
upload |
   
   The new test renders a `.partial().expand()` task with `create_runtime_ti` 
and `ti.render_templates()`, the method the task runner calls, and fails 
without this change. It is skipped before Airflow 3.0 because the fixture needs 
the Task SDK.
   
   `GenericTransfer` in `common.sql` converts `page_size` in a 
`render_template_fields` override in the same way. It is a separate 
distribution and I have not reproduced the problem there, so it is not changed 
here.
   
   closes: #77151
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes — Code Puppy
   
   Generated-by: Code Puppy following [the 
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to