[PR] Replace blocking IO with async IO in AsyncKubernetesHook [airflow]

2023-10-24 Thread via GitHub
functicons opened a new pull request, #35162: URL: https://github.com/apache/airflow/pull/35162 Currently there are blocking IO operations in AsyncKubernetesHook, which are causing triggerer failures or high CPU usage. This change replaces them with async IO. ---

Re: [PR] Replace blocking IO with async IO in AsyncKubernetesHook [airflow]

2023-10-24 Thread via GitHub
functicons commented on PR #35162: URL: https://github.com/apache/airflow/pull/35162#issuecomment-1778027470 > I agree that this needs to be improved, I tried to implement a similar solution in the past but I had a problem with an older version of aiofiles, could you add a unit test fo this

Re: [PR] Replace blocking IO with async IO in AsyncKubernetesHook [airflow]

2023-10-24 Thread via GitHub
functicons commented on PR #35162: URL: https://github.com/apache/airflow/pull/35162#issuecomment-1778201782 Some checks failed with the following error: ``` RuntimeError: By default one of Airflow's dependencies installs a GPL dependency (unidecode). To avoid this dependency set

Re: [PR] Replace blocking IO with async IO in AsyncKubernetesHook [airflow]

2023-10-24 Thread via GitHub
Taragolis commented on PR #35162: URL: https://github.com/apache/airflow/pull/35162#issuecomment-1778249170 > I'm not sure if it's caused by my change? Any help is appreciated! I guess this related to changes in https://github.com/apache/airflow/pull/35099 For some reason `apache-a

Re: [PR] Replace blocking IO with async IO in AsyncKubernetesHook [airflow]

2023-10-25 Thread via GitHub
Taragolis commented on PR #35162: URL: https://github.com/apache/airflow/pull/35162#issuecomment-1780206536 I've checked again, previous error happen because we tried to install Airflow from PyPI with constraints for main branch. If it failed it tried to install airflow without constraint

Re: [PR] Replace blocking IO with async IO in AsyncKubernetesHook [airflow]

2023-10-25 Thread via GitHub
potiuk commented on PR #35162: URL: https://github.com/apache/airflow/pull/35162#issuecomment-1780412471 Hmm. This is an interesting one. The change to use `apache-airflow` has been implemented few days ago - in order to accomodate an interaction with #34729 PR. Honestly, it's super strang

Re: [PR] Replace blocking IO with async IO in AsyncKubernetesHook [airflow]

2023-10-25 Thread via GitHub
potiuk commented on PR #35162: URL: https://github.com/apache/airflow/pull/35162#issuecomment-1780479215 OK. I think I figured out exactly what was going on and have a fix in #35191 that should work in all cases. The change I implemented to make #34729 work with the new pre-installed

Re: [PR] Replace blocking IO with async IO in AsyncKubernetesHook [airflow]

2023-10-26 Thread via GitHub
potiuk commented on PR #35162: URL: https://github.com/apache/airflow/pull/35162#issuecomment-1780709943 Fixed in main. I rebased to see if it helps -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Replace blocking IO with async IO in AsyncKubernetesHook [airflow]

2023-10-26 Thread via GitHub
potiuk commented on PR #35162: URL: https://github.com/apache/airflow/pull/35162#issuecomment-1780950799 > Fixed in main. I rebased to see if it helps It does :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] Replace blocking IO with async IO in AsyncKubernetesHook [airflow]

2023-10-26 Thread via GitHub
hussein-awala commented on PR #35162: URL: https://github.com/apache/airflow/pull/35162#issuecomment-1780979659 My problem was with the CI image too, I thought it was a conflict issue. For the change, I'm using a similar solution in my operators without any issues, so it's safe to mer

Re: [PR] Replace blocking IO with async IO in AsyncKubernetesHook [airflow]

2023-10-26 Thread via GitHub
hussein-awala merged PR #35162: URL: https://github.com/apache/airflow/pull/35162 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@ai