1fanwang opened a new pull request, #71521:
URL: https://github.com/apache/airflow/pull/71521
**TL;DR — operator extra links don't work per attempt** (#71471). After a
retry the button
under attempt 1 opens attempt 2's logs. It hits every bundled provider
linking to per-attempt
logs: EMR, Glue, Databricks, Dataproc, Livy.
Draft **B of three** for the same bug — pick a shape and I'll finish it and
close the others.
| | Approach | Trade |
|---|---|---|
| A | #71518 — a row per attempt in XCom | smallest; core owns a key
convention for every link |
| **B** | **this PR — the link opts in** | **no extra rows when unneeded;
changes the serialized DAG format** |
| C | #71520 — the task state store | nothing about the clear moves; 3.3+
only |
## How
`BaseOperatorLink` gains `keeps_a_link_per_attempt` and `xcom_key_for_try`.
A link that sets
the flag gets a row per attempt, exempt from the retry clear; everything
else behaves as it
does today.
The flag has to survive DAG serialization, and that is the cost. The
api-server never sees a
provider's link class — it rebuilds every link as `XComOperatorLink` from a
`{name: xcom_key}`
mapping carrying a string and nothing else. That mapping now carries
`{"key": ..., "per_attempt": true}` for opted-in links and stays a bare
string otherwise, so
existing serialized DAGs read back unchanged.
## Testing
Read path against a real metastore on 3.2.2, rows present for both attempts,
asking an
opted-in link and a plain one.
<details><summary>Raw output</summary>
```
$ AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=sqlite:///af.db python b_test.py
opted-in try 1 -> https://logs/attempt-1
opted-in try 2 -> https://logs/attempt-2
not opted try 1 -> https://logs/attempt-2
```
Line 3 is the before state on the same path: without the flag, attempt 1
still returns the
latest attempt's URL. Lines 1 and 2 are the same read with it set.
Key derivation:
```
$ python -c "from airflow.serialization.definitions.operatorlink import
XComOperatorLink; ..."
default try1 -> _link_X | try2 -> _link_X
opted-in try1 -> _link_attempt_1__link_X | try2 -> _link_attempt_2__link_X
```
Serialization round-trip, the risky part:
```
encoded: {'plain': '_link_P', 'opted': {'key': '_link_O', 'per_attempt':
True}}
plain: key=_link_P per_attempt=False (legacy string, unchanged)
opted: key=_link_O per_attempt=True
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
