llamageddon83 opened a new issue, #32928:
URL: https://github.com/apache/airflow/issues/32928
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
We are running Airflow on EKS with version 2.5.3. Airflow has been
experiencing progr
llamageddon83 commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1815544361
Hi all,
Configuring `AIRFLOW__SCHEDULER__NUM_RUNS` has fixed the problem for us.
Thank you
--
This is an automated message from the Apache Git Service.
To re
droppoint commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1820413530
Hi, everyone!
I think I found the root cause of the problem.
Short answer: The
[KubernetesExecutor._adopt_completed_pods](https://github.com/apache/airflow/blob/main
potiuk commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1820957816
I pinged at the `#development` channel of Airflow's Slack as we are now
gearing up for 2.8.0 release that might be a good opportunity to have some
maintainers to take a close look.
droppoint commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1821006647
@potiuk Thank you. Should I submit a PR with the _adopt_completed_pods
removal or is it better to wait for the maintainers decision on how to fix this
problem?
--
This is an a
potiuk commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1821014882
Worst-case PR will be closed and another one will supersede it - and
everyone will learn from the comments in the first one :)
--
This is an automated message from the Apache Git
github-actions[bot] commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1841859469
This issue has been automatically marked as stale because it has been open
for 14 days with no response from the author. It will be closed in next 7 days
if no further
dirrao commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1864449351
Hi @llamageddon83, @droppoint,
We are facing this similar issue more often. Here is the another ticket for
the same #35675. I have created the MR for avoiding the executor open s
droppoint commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1864543583
@dirrao This is cool, but I already submitted PR #35800 almost a month ago.
I think we are fixing different problems. Your PR addresses the issue when
adoption is performed on a
potiuk commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1864638111
> P.S. Can you share the secret of how you managed to get a review from the
maintainer so fast?
I can share a secret. Be kind, be considerate, but also ... be persistent.
The
potiuk commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1864638561
I will ping in `development` again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to t
potiuk closed issue #32928: Airflow progressive slowness
URL: https://github.com/apache/airflow/issues/32928
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-m
potiuk commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1864652858
Reopening as the #35800 seems to also address part of the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and
ephraimbuddy commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1864656976
Did the PR actually fixed this issue? I think we need to benchmark it before
closing. We have seen this issue with celery and the PR was on kubernetes
--
This is an automat
dirrao commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1864668705
I see multiple issues are reported in the same issue. We might need to
create separate issue for Celery executor.
--
This is an automated message from the Apache Git Service.
To r
potiuk commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1864794389
Yeah. I think there could be many similar behaviours for various reasons. I
think we should keep on getting info from people reporting similar issues after
upgrading to latest versi
dirrao commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1864809538
I have created a new issue #36335 for celery executor. Let's keep this for
Kubernetes executor.
--
This is an automated message from the Apache Git Service.
To respond to the mes
dirrao commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1867826875
@llamageddon83
We have faced a similar issue in production, identified the root cause, and
provided the fix in #36240. If you are looking for an immediate fix, then you
can try
dirrao commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1870790534
Hi @llamageddon83,
Kubernets executor slots leak fix is available in 7.12.0 version. You can
try and see if this fix your issue.
https://airflow.apache.org/docs/apache-airfl
ephraimbuddy commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1875673643
FYI, I updated `[scheduler]max_tis_per_dag` to 512 in 2.8.0 and saw good
improvement in the number of tasks ran per minute
--
This is an automated message from the Apache Gi
github-actions[bot] commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1933149224
This issue has been closed because it has not received response from the
issue author.
--
This is an automated message from the Apache Git Service.
To respond to the
github-actions[bot] closed issue #32928: Airflow progressive slowness
URL: https://github.com/apache/airflow/issues/32928
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To uns
github-actions[bot] commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1771870798
This issue has been automatically marked as stale because it has been open
for 14 days with no response from the author. It will be closed in next 7 days
if no further
sa2413 commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1782211535
any update on this issue? facing this on multiple clusters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use t
potiuk commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1783424675
No, but if someone would like to spend time on trying to analyse it it would
be great. Until then - if you experience it, I recommend restarting airflow
periodically. Seems like sup
harshg0910 commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1783709930
Hi @potiuk and @ephraimbuddy, I see "Airflow 2.7.3" milestone added to this
bug, I was under the impression that it is being fixed in next bugfix release.
Is it not the case (so
potiuk commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1783797115
> Hi @potiuk and @ephraimbuddy, I see "Airflow 2.7.3" milestone added to
this bug, I was under the impression that it is being fixed in next bugfix
release. Is it not the case (sorr
potiuk commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1783915309
I prepared pr #35245 and started proposal on devlist to add description on
the process/approach we are using. See
https://lists.apache.org/thread/05njmmqvwl0gn20f2go9d420xhzptrw2 -
llamageddon83 commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1797040724
Hi all,
Thank you to everyone who provided input and to the Airflow team for
following up on this. While going through some old posts from @potiuk on
stackoverflow in
adrianrego commented on issue #32928:
URL: https://github.com/apache/airflow/issues/32928#issuecomment-1748783239
I'm also seeing this issue on our setup...Airflow 2.7.1, KubernetesExecutor.
Running 3 scheduler pods. Looks like things start going downhill after 5
days. Restarting the sch
30 matches
Mail list logo
