uranusjr opened a new pull request, #77159: URL: https://github.com/apache/airflow/pull/77159
When running a mapped task, the worker needs each expand argument's length to find the element for its map index. Every argument is already resolved at that point and reports its own length, but the worker instead ran a separate per-type calculation over the XCom argument tree. That calculation dates from Airflow 2, where lengths came from database queries and each XCom argument type needed its own. #46032 replaced the queries with `len(resolved_value)` but kept the dispatch, handing the whole resolved value to every node. This works out to `len(resolved_value)` for plain, `map()` and `zip()` arguments, but `concat()` adds it once per part. When a `concat()` argument is not the first one declared, the inflated length selects an out-of-range index, so `pull.expand(x=[10, 20], y=a.concat(b))` fails with IndexError. The calculation also multiplied a plain argument's length by `upstream_map_indexes[task_id]`. That field was unset when this was written, later reused for `xcom_pull`, and is no longer sent by current servers. An old server only fills it when the task and upstream share a mapped task group, which an expand argument never does, so the multiplier was always 1. Using the resolved value directly keeps the behavior of other argument types, fixes `concat()`, and leaves one source of truth for map lengths on the worker. The scheduler computes lengths without loading XCom values, so it keeps its own per-type calculation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
