uranusjr opened a new pull request, #77159:
URL: https://github.com/apache/airflow/pull/77159

   When running a mapped task, the worker needs each expand argument's length 
to find the element for its map index. Every argument is already resolved at 
that point and reports its own length, but the worker instead ran a separate 
per-type calculation over the XCom argument tree.
   
   That calculation dates from Airflow 2, where lengths came from database 
queries and each XCom argument type needed its own. #46032 replaced the queries 
with `len(resolved_value)` but kept the dispatch, handing the whole resolved 
value to every node. This works out to `len(resolved_value)` for plain, `map()` 
and `zip()` arguments, but `concat()` adds it once per part. When a `concat()` 
argument is not the first one declared, the inflated length selects an 
out-of-range index, so `pull.expand(x=[10, 20], y=a.concat(b))` fails with 
IndexError.
   
   The calculation also multiplied a plain argument's length by 
`upstream_map_indexes[task_id]`. That field was unset when this was written, 
later reused for `xcom_pull`, and is no longer sent by current servers. An old 
server only fills it when the task and upstream share a mapped task group, 
which an expand argument never does, so the multiplier was always 1.
   
   Using the resolved value directly keeps the behavior of other argument 
types, fixes `concat()`, and leaves one source of truth for map lengths on the 
worker. The scheduler computes lengths without loading XCom values, so it keeps 
its own per-type calculation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to