Re: [I] Last DAG RUN not showing up on DAG list page in UI [airflow]
pierrejeambrun closed issue #49135: Last DAG RUN not showing up on DAG list page in UI URL: https://github.com/apache/airflow/issues/49135 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Last DAG RUN not showing up on DAG list page in UI [airflow]
pierrejeambrun commented on issue #49135: URL: https://github.com/apache/airflow/issues/49135#issuecomment-2909925682 I'm going to take a look at this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Last DAG RUN not showing up on DAG list page in UI [airflow]
tirkarthi commented on issue #49135:
URL: https://github.com/apache/airflow/issues/49135#issuecomment-2894859924
An idea had in mind was to pass the the list of dag_id rendered in the page
from API to `dagIds` but with 100 dags per page I am afraid this might hit some
browser limits.
https://stackoverflow.com/questions/812925/what-is-the-maximum-possible-length-of-a-query-string
```patch
diff --git a/airflow-core/src/airflow/ui/src/queries/useDags.tsx
b/airflow-core/src/airflow/ui/src/queries/useDags.tsx
index d4f58dc985..653b9c53e7 100644
--- a/airflow-core/src/airflow/ui/src/queries/useDags.tsx
+++ b/airflow-core/src/airflow/ui/src/queries/useDags.tsx
@@ -51,10 +51,12 @@ export const useDags = (
} = useDagsServiceRecentDagRuns(
{
...runsParams,
+ dagIds: data?.dags?.map((dag) => dag.dag_id),
dagRunsLimit: 14,
},
undefined,
{
+ enabled: Boolean(data),
refetchInterval: (query) =>
query.state.data?.dags.some(
(dag) => !dag.is_paused && dag.latest_dag_runs.some((dr) =>
isStatePending(dr.state)),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] Last DAG RUN not showing up on DAG list page in UI [airflow]
jsjasonseba commented on issue #49135: URL: https://github.com/apache/airflow/issues/49135#issuecomment-2894668735 > [@jsjasonseba](https://github.com/jsjasonseba) how are doing on this? My initial PR haven't perfectly solved the issue. There were some discussions in the PR finding that the root cause of the issue is more complex than what was initally thought. I haven't got the chance to rethink the solution. I'm fine if there's anyone willing to take this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Last DAG RUN not showing up on DAG list page in UI [airflow]
vatsrahul1001 commented on issue #49135: URL: https://github.com/apache/airflow/issues/49135#issuecomment-2889693696 @jsjasonseba how are doing on this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Last DAG RUN not showing up on DAG list page in UI [airflow]
jsjasonseba commented on issue #49135: URL: https://github.com/apache/airflow/issues/49135#issuecomment-2809576187 > Indeed, taking a closer look I don't think limits will match, unless we pass `dag_runs_limit` to `1` maybe ? That will just take 1 dag_run per dag_id, and therefore the final `limit` applied should be the correct one, might be worth a try. Right! This should work. Let me implement this fix too. Currently it take dag_runs_limit of 14, but 1 should be enough -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Last DAG RUN not showing up on DAG list page in UI [airflow]
pierrejeambrun commented on issue #49135: URL: https://github.com/apache/airflow/issues/49135#issuecomment-2809566654 Indeed, taking a closer look I don't think limits will match, unless we pass `dag_runs_limit` to `1` maybe ? That will just take 1 dag_run per dag_id, and therefore the final `limit` applied should be the correct one, might be worth a try. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Last DAG RUN not showing up on DAG list page in UI [airflow]
jsjasonseba commented on issue #49135: URL: https://github.com/apache/airflow/issues/49135#issuecomment-2801162739 > for `recent_dag_runs` there are two limits `limit` `dag_runs_limit`. `limit` is on the dags. CMIIW, but the `limit` in `recent_dag_runs` is applied on data grouped by of DAG Run not DAG. So, I think it does not match with limit in /dags call either -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Last DAG RUN not showing up on DAG list page in UI [airflow]
jsjasonseba commented on issue #49135: URL: https://github.com/apache/airflow/issues/49135#issuecomment-280114 > I don't think we want to include that information in the public API. The idea was to keep `recent_dag_runs` in the `ui` api so we don't have to support any versioning for it and can modify it based on the UI need. > > I think that propagating the appropriate order_by should solve the issue. Noted, will implement with this approach -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Last DAG RUN not showing up on DAG list page in UI [airflow]
pierrejeambrun commented on issue #49135: URL: https://github.com/apache/airflow/issues/49135#issuecomment-2801116582 for `recent_dag_runs` there are two limits `limit` `dag_runs_limit`. `limit` is on the dags. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Last DAG RUN not showing up on DAG list page in UI [airflow]
pierrejeambrun commented on issue #49135: URL: https://github.com/apache/airflow/issues/49135#issuecomment-2801114705 I don't think we want to include that information in the public API. The idea was to keep `recent_dag_runs` in the `ui` api so we don't have to support any versioning for it and can modify it based on the UI need. I think that propagating the appropriate order_by should solve the issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Last DAG RUN not showing up on DAG list page in Ui [airflow]
jsjasonseba commented on issue #49135: URL: https://github.com/apache/airflow/issues/49135#issuecomment-2798849289 Agreed, here is my idea. We could add a boolean parameter `include_last_dag_run` to the GET /dags API, when enabled, includes the last dag run date pre-joined from backend. This would eliminate the need for the frontend to perform an additional join. Please let me know if there are specific concerns regarding this approach @pierrejeambrun @bbovenzi -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Last DAG RUN not showing up on DAG list page in Ui [airflow]
vatsrahul1001 commented on issue #49135: URL: https://github.com/apache/airflow/issues/49135#issuecomment-2798818856 In my opinion, it would be better to handle this in the backend. However, I'm happy to go with whatever @pierrejeambrun or @bbovenzi recommends. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Last DAG RUN not showing up on DAG list page in Ui [airflow]
jsjasonseba commented on issue #49135: URL: https://github.com/apache/airflow/issues/49135#issuecomment-2798789424 Hi @tirkarthi and @vatsrahul1001, I have identified the root cause of the issue. It is caused by the frontend logic where it gets a list of DAG from `api/v2/dags?limit=100&offset=0&order_by=-last_run_start_date`. However, since this API does not offer information about last DAG run, another call is needed which is `/ui/dags/recent_dag_runs?dag_runs_limit=14&limit=100&offset=100`. Then, a join is performed on frontend side. However, the recent_dag_runs call does not follow the ordering of the dag list call. So, what ends up happening is, by default dag list call is ordered by last run start date (it fetches top 100 DAG ordered by last run start date), and the recent dag run call always orders by last run run_after (fetches top 100 DAG with last run run_after), so it ends up returning different set of DAGs and the join have missing last run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Last DAG RUN not showing up on DAG list page in Ui [airflow]
jsjasonseba commented on issue #49135: URL: https://github.com/apache/airflow/issues/49135#issuecomment-2798791108 I was able to reproduce a similar case when ordering using DAG ID in DAG list page. My DAGs on 2nd page are missing their last run (need to have 100+ DAG so we have page 2). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Last DAG RUN not showing up on DAG list page in Ui [airflow]
jsjasonseba commented on issue #49135: URL: https://github.com/apache/airflow/issues/49135#issuecomment-2798792483 @vatsrahul1001 I'm happy to take this, but I would like to discuss some options on resolving this. We can either fix this in the backend side by doing joins on the backend, or from the frontend side by making sure the recent_dag_runs call respect the DAG list ordering. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Last DAG RUN not showing up on DAG list page in Ui [airflow]
vatsrahul1001 commented on issue #49135: URL: https://github.com/apache/airflow/issues/49135#issuecomment-2798791969 @jsjasonseba would you like to take this up? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Last DAG RUN not showing up on DAG list page in Ui [airflow]
vatsrahul1001 commented on issue #49135: URL: https://github.com/apache/airflow/issues/49135#issuecomment-2798512966 Yes I am able to repro this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Last DAG RUN not showing up on DAG list page in Ui [airflow]
vatsrahul1001 commented on issue #49135: URL: https://github.com/apache/airflow/issues/49135#issuecomment-2798407286 Yeah not able to reproduce now with same steps. I will take a look again to repro -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Last DAG RUN not showing up on DAG list page in Ui [airflow]
tirkarthi commented on issue #49135: URL: https://github.com/apache/airflow/issues/49135#issuecomment-2798406611 Not able to reproduce this with http://localhost:8000/ui/dags/recent_dag_runs?dag_runs_limit=14&limit=15&offset=0 endpoint returning response as expected in the table mode. @vatsrahul1001 Is this consistently reproducible? cc: @bbovenzi @pierrejeambrun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
