shahar1 opened a new pull request, #77211:
URL: https://github.com/apache/airflow/pull/77211

   Partition status is visible today only per consuming Dag (pending runs in 
the Dags list) and as a "Partition Key" line on an asset event. Nothing answers 
"which partitions of this asset are materialized, in progress or failed". A 
failed or running producer run leaves no event, so it is invisible from the 
asset page.
   
   This adds a read-only **Partitions** tab to the asset page 
(`/assets/:assetId/partitions`) and one UI-internal route, `GET 
/ui/assets/{asset_id}/partitions`. There is no public REST API change, no 
migration, and no Task SDK or scheduler change.
   
   ## What the tab shows
   
   - A count per status, which also toggles a status filter.
   - A day grid, only when every key on the page is a calendar day 
(`YYYY-MM-DD`, with or without `T00:00:00`).
   - A table: partition key, state, last producer run, last asset event, and 
consumer runs (Dag runs that consumed the last event).
   
   ## Screenshots
   
   From `breeze start-airflow`, with a daily partitioned producer and seeded 
demo runs.
   
   ### Before
   
   The asset page on `main`, same data. It has two tabs, and a partition 
appears only as a "Partition Key" line on each event.
   
   **>>> DRAG IMAGE HERE: 1-before-wide-light.png (Asset page before the 
change, widened panel, light mode) <<<**
   <img width="1440" height="900" alt="1-before-wide-light" 
src="https://github.com/user-attachments/assets/a5caf1df-7fa2-46af-a509-558da83ffea0";
 />
   
   **>>> DRAG IMAGE HERE: 2-before-wide-dark.png (Asset page before the change, 
widened panel, dark mode) <<<**
   <img width="1440" height="900" alt="2-before-wide-dark" 
src="https://github.com/user-attachments/assets/0d9116fa-ddcd-4b03-acbd-85f52e74e15f";
 />
   
   **>>> DRAG IMAGE HERE: 3-before-default-light.png (Asset page before the 
change, default panel width, light mode) <<<**
   <img width="1440" height="900" alt="3-before-default-light" 
src="https://github.com/user-attachments/assets/ee0a51e6-eb6f-4a2e-a4ae-f5abac9f1b63";
 />
   
   ### After
   
   Widened details panel, light and dark:
   
   **>>> DRAG IMAGE HERE: 4-partitions-wide-light.png (Partitions tab, widened 
panel, light mode) <<<**
   <img width="1440" height="900" alt="4-partitions-wide-light" 
src="https://github.com/user-attachments/assets/40a3c8de-113b-480c-831b-5ec6e87e59a6";
 />
   
   **>>> DRAG IMAGE HERE: 5-partitions-wide-dark.png (Partitions tab, widened 
panel, dark mode) <<<**
   <img width="1440" height="900" alt="5-partitions-wide-dark" 
src="https://github.com/user-attachments/assets/853a05a6-7ae3-4aa0-a3bc-76cde348375e";
 />
   
   Default panel width (the table scrolls sideways):
   
   **>>> DRAG IMAGE HERE: 6-partitions-default-light.png (Partitions tab, 
default panel width, light mode) <<<**
   <img width="1440" height="900" alt="6-partitions-default-light" 
src="https://github.com/user-attachments/assets/4e7f0524-39be-42b1-bed2-4d504d9fa0ff";
 />
   
   Status filter switched on for In Progress and Failed:
   
   **>>> DRAG IMAGE HERE: 7-partitions-filtered-default-dark.png (Partitions 
tab with the status filter on, dark mode) <<<**
   <img width="1440" height="900" alt="7-partitions-filtered-default-dark" 
src="https://github.com/user-attachments/assets/8642f64b-c39e-44ec-878b-2c68a8a4ef29";
 />
   
   Keys that are not calendar days, so no day grid:
   
   **>>> DRAG IMAGE HERE: 8-partitions-freeform-default-light-panel.png 
(Partitions tab for an asset with free-form keys) <<<**
   <img width="405" height="823" 
alt="8-partitions-freeform-default-light-panel" 
src="https://github.com/user-attachments/assets/bcba8a73-a7e9-4df0-b334-931c1a195f9f";
 />
   
   ## How status is derived
   
   For each key, `E` is the latest event and `R` the newest attempt among the 
runs of Dags that declare the asset as an outlet. A run belongs to its own 
`partition_key`.
   
   - `R` queued or running, and no `E` since the attempt began: **In Progress**
   - `R` failed, and no `E` since the attempt began: **Failed**
   - otherwise, `E` exists: **Materialized**
   - otherwise the key is not listed
   
   An attempt begins at `queued_at`, which clearing resets. Runs and events of 
Dags the caller may not read are left out before aggregation, so neither rows 
nor counts reveal them.
   
   ## Known limits
   
   - There is no "missing" state: an asset does not declare its partitions, so 
an absent key cannot be told from a key that never existed.
   - Attribution is per Dag run, not per task. A run that fails before emitting 
anything is tied to its own key, even if its task would have emitted another 
key. A run that emitted the asset only under other keys is left out.
   - Each request scans the asset's event keys and its producer Dags' runs, for 
the counts and again for the page. `dag_run.partition_key` has no index.
   
   ## Testing
   
   - `uv run --project airflow-core pytest 
airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_assets.py`: 71 
passed, 30 of them for the new route.
   - `pnpm lint`: clean. `pnpm test --maxWorkers=6`: 2,450 passed. With the 
default worker count, three test files unrelated to this change fail on my 
machine, and they fail the same way on the base commit.
   - `prek run --from-ref main --stage pre-commit`: passed.
   - Run in `breeze start-airflow` with a daily partitioned producer and seeded 
runs; checked in light and dark mode, at the default and a widened details 
panel.
   
   Not run: the manual prek stage, `breeze verify`, PostgreSQL and MySQL 
backends, and the Playwright e2e suite.
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes — Claude Code (Fable 5.2)
   
   Generated-by: Claude Code (Fable 5.2) following [the 
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to