llamageddon83 commented on issue #32928: URL: https://github.com/apache/airflow/issues/32928#issuecomment-1698049513
Hi @potiuk Please find my answers in line below: Is it possible that you do check which tasks are in "running" state for some time when it happens and maybe try to find some correlated logs from scheduler and task executtion - and ideally also find the logs from another task instance of the same task that was not found in "running" so that we could compare them and see the difference? > Yes we have spent quite a bit of time trying to compare logs from different tasks, logs of same tasks from different executions. But there is no oddity we could spot. As far as `running` state goes, since this happens very sporadically, we don't know what happens to the task state. Whenever we take a look, the tasks have completed and are no longer in the `running` state. I preseume that there is some kind of race happenin that will somehow skip removal of the task even if it has been completed. - but seeing the logs and comparing them could narrow down the searrch an might allow us to come up with a plausible hypothesis. > Sorry I wish I could help. We have spent weeks trying to narrow it down to some DAG implementation, some kind of pattern we may have.. but we got nothing.. Question - do you run deferrable tasks / triggerer ? Maybe somehow there is a problem with deferrable code ? (wild guessing now after looking into some possible paths). > Yes, we have quite a few of them. The deferrable tasks run in a never ending loop. I know you are very busy and go through many discussion, but if you remember, I previously mentioned how we use triggers for streaming [here](https://github.com/apache/airflow/discussions/31074). The triggers schedule is stacked and they ramp up to their max_active_runs. So they will start one by one with a few minutes interval in between. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
