abhijeets25012-tech commented on code in PR #76935:
URL: https://github.com/apache/airflow/pull/76935#discussion_r4547521749
##########
task-sdk/src/airflow/sdk/definitions/xcom_arg.py:
##########
@@ -465,6 +465,9 @@ class _ZipResult(Sequence):
@staticmethod
def _get_or_fill(container: Sequence | dict, index: Any, fillvalue: Any)
-> Any:
+ if isinstance(container, dict):
+ # dict upstreams follow the same (key, value) convention as map()
+ container = list(container.items())
Review Comment:
`_get_or_fill` runs for every index, so this rebuilds the items list for
each element. Zipping a 20k-key dict with a 20k list takes about 23s here;
building the items list once in `ZipXComArg.resolve()` gives the same result in
about 0.06s.
##########
task-sdk/src/airflow/sdk/definitions/xcom_arg.py:
##########
@@ -535,6 +538,9 @@ def __getitem__(self, index: Any) -> Any:
break
if i >= (curlen := len(value)):
i -= curlen
+ elif isinstance(value, dict):
+ # dict upstreams follow the same (key, value) convention as
map()
+ return next(itertools.islice(value.items(), i, None))
Review Comment:
Same quadratic shape: `islice` walks to `i` on every access, so iterating a
20k-key dict takes about 1.6s, versus about 0.01s if the items are materialised
once in `ConcatXComArg.resolve()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
