I do not have the use case and prefer explicit pinning to the task where
I want XCom from. But maybe my/our use cases are structured and
repeatable that the questions does not hold.
So if there is the demand for a non-deterministic retrieval of the
"OLDEST" / "NEWEST" (single) entry I think using the constant classes is
a good approach not to overload the function signature. If this is well
documented in PyDoc then it is accessible. Would be OK for me.
Otherwise a more generic solution could be providing an API which
accepts a list of "candidates" to poll meta data and get a lis and this
list is including dates. Then some logic could sort by date and retrieve
whatever es desired... if there are more niece use cases this might be
an alternative. But the proposal is lightweight.
On 15.04.26 17:03, Jarek Potiuk wrote:
I'd wait for others to comment. I want to see what others think.
On Wed, Apr 15, 2026 at 4:51 PM Foldvari, Gyorgy via dev <
[email protected]> wrote:
If xcom_pull(task_ids=XCom.ANY, key=”sample”) would return with a list,
that would not be the same result what was produced by Airflow 2. Airflow 2
returns with the most recent value and not with a list.
I am wondering what would make more sense:
1. Return with one value, the most recent one (on None) instead of a
list, copying Airflow 2 behavior.
2. Return with a list but also defining a new optional parameter to
override this behavior (XCom.FIRST_OLDEST, XCom.FIRST_NEWEST).
I am struggling with coming up a valid use case where getting back a list
of values without any further information would make sense. So probably we
should go with the simpler first option.
Information Classification: GENERAL
On 2026/04/15 14:35:35 Jarek Potiuk wrote:
What do you think about this solution?
ti.xcom_pull accepting ‘*’ or rather a sentinel value as task_ids to
explicitly say that ANY task_id should be considered. The result would be
all the values pushed to xcom by any task in that dag run as a list,
sorted
by XCOM timestamp.
ti.xcom_pull(task_ids= ANY, key=”example”)
I'd rather (if others see a value for it) have some explicit sentinel
(XCom.FiRST_OLDEST, XCom.FIRST_NEWEST - or whatever makes sense for
those).
The only challenge is how to use this in classic Operators’ templated
fields.
Those sentinels could be easily exposed to JINJA and predefined there.
It might of course be over-the-top, and some might say "ugly" - but if it
makes it easier for people to migrate to 3 and would not make it more
difficult for maintenance, I am all for it.
J.
On Wed, Apr 15, 2026 at 4:22 PM Foldvari, Gyorgy via dev <
[email protected]> wrote:
What do you think about this solution?
ti.xcom_pull accepting ‘*’ or rather a sentinel value as task_ids to
explicitly say that ANY task_id should be considered. The result would
be
all the values pushed to xcom by any task in that dag run as a list,
sorted
by XCOM timestamp.
ti.xcom_pull(task_ids= ANY, key=”example”)
The only challenge is how to use this in classic Operators’ templated
fields.
Information Classification: GENERAL
On 2026/04/15 13:34:58 Jarek Potiuk wrote:
This was ambiguous regarding the value you wanted to retrieve and
the
intended change.
While I see why we cannot (should not) bring back the "exact"
behaviour,
I
think we could - potentially at least - make it non-ambiguous and
predictable by defining the semantics and ordering and selecting the
first
one. And - I think - we do not **have** to bring it back in this
particular
method. Maybe we can just a new method call with defined ordering and
semantics - behaving predictably, similar to Airflow 2—and clearly
defined
semantically. That would at least give people an easier way to
migrate?
While I think the cat if out-of-the bag and we cannot truly revert
the
change (because that would again potentially affect 3.0 - 3.2
users) -
but
we could at least make it easier for people to cope with it without
too
much hassle while waiting for the task state to be available in this
particular case?
Just a thought I had - ... listen to your users and do things easier
for
them - without breaking our SemVer promises.
J.
On Wed, Apr 15, 2026 at 3:20 PM Foldvari, Gyorgy via dev <
[email protected]> wrote:
I do not want to use XCom for managing task state. It was just a
very
simple - and seemingly misleading - example to explain the original
behavior.
I gave details about valid use cases and issues caused by this
change
in
the behavior, in my original post. Those are not addressed by the
AIP
you
are referring to.
But that AIP would definitely address another valid use case what
I am
missing especially for implementing stateful sensors, so I really
hope
it
goes through and gets implemented.
Information Classification: GENERAL
-----Original Message-----
From: Ash Berlin-Taylor <[email protected]>
Sent: Wednesday, April 15, 2026 14:49
To: [email protected]
Subject: Re: [DISCUSS] Reconsidering `xcom_pull(task_ids=None)`
behavior
change in Airflow 3
[You don't often get email from [email protected]. Learn why this is
important at https://aka.ms/LearnAboutSenderIdentification ]
CAUTION: This email originated from outside of the organization.
Do not
click links or open attachments unless you recognize the sender and
know
the content is safe.
This was ambiguous as to what value you wanted to get, and an
intended
change.
If you want this sort of behaviour, then you probably want to look
at
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-103%3A+Task+State+Management
which provides a dedicated way to manage state without many of the
quirks
of XCom interface as it stands today. Reading between the lines, I
think
this API describes what you want?
-ash
On 15 Apr 2026, at 12:45, Foldvari, Gyorgy via dev <
[email protected]> wrote:
I see where the confusion is coming from, it is my mistake. Sorry
about
that.
To clarify, I am taking about the use case where the key
parameter is
passed but the task_ids parameter is not or it is None.
-----Original Message-----
From: Foldvari, Gyorgy via dev <[email protected]>
Sent: Wednesday, April 15, 2026 13:42
To: [email protected]
Cc: Foldvari, Gyorgy <[email protected]>
Subject: RE: Re: [DISCUSS] Reconsidering
`xcom_pull(task_ids=None)`
behavior change in Airflow 3
CAUTION: This email originated from outside of the organization.
Do
not
click links or open attachments unless you recognize the sender and
know
the content is safe.
The original behavior is to return the most recent value put by
any
upstream task of the same run. Not all the values, only the recent
one.
Supposing that there are multiple tasks pushing values to XCom in
this
order:
Task1: ti.xcom_push(key="example", value=1)
Task2: ti.xcom_push(key="example", value=2)
Task3: ti.xcom_push(key="example", value=3) Then in a downstream
task
ti.comm_pull(key="example") returns 3.
I do not propose to change this behavior.
On 2026/04/14 16:09:05 Daniel Standish via dev wrote:
So the behavior before would be that it would return all xcom
values
that were emitted from the present run?
________________________________
Information regarding MSCI's processing of personal data may be
found
at http://www.msci.com/privacy-pledge. This email message and
any
attachments
are for the sole use of the intended recipients and may contain
proprietary and/or confidential information which may be
privileged
or
otherwise protected from disclosure. Any unauthorized review,
use,
disclosure or distribution is prohibited. All rights and
remedies are
reserved. If you are not an intended recipient, please contact
the
sender by reply email and destroy the original message and any
copies
of the message as well as any attachments to the original
message.
Local registered entity information:
https://www.msci.com/local-registered-entities
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
________________________________
Information regarding MSCI’s processing of personal data may be
found
at
http://www.msci.com/privacy-pledge. This email message and any
attachments are for the sole use of the intended recipients and may
contain
proprietary and/or confidential information which may be
privileged or
otherwise protected from disclosure. Any unauthorized review, use,
disclosure or distribution is prohibited. All rights and remedies
are
reserved. If you are not an intended recipient, please contact the
sender
by reply email and destroy the original message and any copies of
the
message as well as any attachments to the original message. Local
registered entity information:
https://www.msci.com/local-registered-entities
________________________________
Information regarding MSCI’s processing of personal data may be found
at
www.msci.com/privacy-pledge. This email message and any attachments
are
for the sole use of the intended recipients and may contain proprietary
and/or confidential information which may be privileged or otherwise
protected from disclosure. Any unauthorized review, use, disclosure or
distribution is prohibited. All rights and remedies are reserved. If
you
are not an intended recipient, please contact the sender by reply
email and
destroy the original message and any copies of the message as well as
any
attachments to the original message. Local registered entity
information:
https://www.msci.com/local-registered-entities
________________________________
Information regarding MSCI’s processing of personal data may be found at
www.msci.com/privacy-pledge. This email message and any attachments are
for the sole use of the intended recipients and may contain proprietary
and/or confidential information which may be privileged or otherwise
protected from disclosure. Any unauthorized review, use, disclosure or
distribution is prohibited. All rights and remedies are reserved. If you
are not an intended recipient, please contact the sender by reply email and
destroy the original message and any copies of the message as well as any
attachments to the original message. Local registered entity information:
https://www.msci.com/local-registered-entities
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
