Re: [PR] Always use standalone DAG processor in breeze [airflow]
potiuk commented on code in PR #45699: URL: https://github.com/apache/airflow/pull/45699#discussion_r1919268594 ## dev/breeze/src/airflow_breeze/params/shell_params.py: ## @@ -597,7 +596,7 @@ def env_variables_for_docker_commands(self) -> dict[str, str]: _set_var(_env, "SKIP_SSH_SETUP", self.skip_ssh_setup) _set_var(_env, "SQLITE_URL", self.sqlite_url) _set_var(_env, "SSH_PORT", None, SSH_PORT) -_set_var(_env, "STANDALONE_DAG_PROCESSOR", self.standalone_dag_processor) +_set_var(_env, "STANDALONE_DAG_PROCESSOR", "true") Review Comment: > Well, that was pretty easy: https://github.com/apache/airflow/pull/45723 Yeah, I never told it was difficult ;) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Always use standalone DAG processor in breeze [airflow]
potiuk commented on PR #45699: URL: https://github.com/apache/airflow/pull/45699#issuecomment-2596962223 Closing after merging https://github.com/apache/airflow/pull/45723 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Always use standalone DAG processor in breeze [airflow]
potiuk closed pull request #45699: Always use standalone DAG processor in breeze URL: https://github.com/apache/airflow/pull/45699 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Always use standalone DAG processor in breeze [airflow]
potiuk commented on code in PR #45699: URL: https://github.com/apache/airflow/pull/45699#discussion_r1919208206 ## dev/breeze/src/airflow_breeze/params/shell_params.py: ## @@ -597,7 +596,7 @@ def env_variables_for_docker_commands(self) -> dict[str, str]: _set_var(_env, "SKIP_SSH_SETUP", self.skip_ssh_setup) _set_var(_env, "SQLITE_URL", self.sqlite_url) _set_var(_env, "SSH_PORT", None, SSH_PORT) -_set_var(_env, "STANDALONE_DAG_PROCESSOR", self.standalone_dag_processor) +_set_var(_env, "STANDALONE_DAG_PROCESSOR", "true") Review Comment: Fix for the fastapi annoyance https://github.com/apache/airflow/pull/45724 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Always use standalone DAG processor in breeze [airflow]
jedcunningham commented on code in PR #45699: URL: https://github.com/apache/airflow/pull/45699#discussion_r1919205506 ## dev/breeze/src/airflow_breeze/params/shell_params.py: ## @@ -597,7 +596,7 @@ def env_variables_for_docker_commands(self) -> dict[str, str]: _set_var(_env, "SKIP_SSH_SETUP", self.skip_ssh_setup) _set_var(_env, "SQLITE_URL", self.sqlite_url) _set_var(_env, "SSH_PORT", None, SSH_PORT) -_set_var(_env, "STANDALONE_DAG_PROCESSOR", self.standalone_dag_processor) +_set_var(_env, "STANDALONE_DAG_PROCESSOR", "true") Review Comment: Well, that was pretty easy: #45723 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Always use standalone DAG processor in breeze [airflow]
jedcunningham commented on code in PR #45699: URL: https://github.com/apache/airflow/pull/45699#discussion_r1919194891 ## dev/breeze/src/airflow_breeze/params/shell_params.py: ## @@ -597,7 +596,7 @@ def env_variables_for_docker_commands(self) -> dict[str, str]: _set_var(_env, "SKIP_SSH_SETUP", self.skip_ssh_setup) _set_var(_env, "SQLITE_URL", self.sqlite_url) _set_var(_env, "SSH_PORT", None, SSH_PORT) -_set_var(_env, "STANDALONE_DAG_PROCESSOR", self.standalone_dag_processor) +_set_var(_env, "STANDALONE_DAG_PROCESSOR", "true") Review Comment: Let me see if I can get the 2 way flag going. But standalone was around since 2.3, which should be plenty of backcompat imo. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Always use standalone DAG processor in breeze [airflow]
potiuk commented on code in PR #45699: URL: https://github.com/apache/airflow/pull/45699#discussion_r1919190756 ## dev/breeze/src/airflow_breeze/params/shell_params.py: ## @@ -597,7 +596,7 @@ def env_variables_for_docker_commands(self) -> dict[str, str]: _set_var(_env, "SKIP_SSH_SETUP", self.skip_ssh_setup) _set_var(_env, "SQLITE_URL", self.sqlite_url) _set_var(_env, "SSH_PORT", None, SSH_PORT) -_set_var(_env, "STANDALONE_DAG_PROCESSOR", self.standalone_dag_processor) +_set_var(_env, "STANDALONE_DAG_PROCESSOR", "true") Review Comment: (BTW - it IS a bit broken now - because it also starts fast-api today when yoy run old version - I wanted to fix it as well though, because it's annoying to see it fail... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Always use standalone DAG processor in breeze [airflow]
potiuk commented on code in PR #45699: URL: https://github.com/apache/airflow/pull/45699#discussion_r1919188984 ## dev/breeze/src/airflow_breeze/params/shell_params.py: ## @@ -597,7 +596,7 @@ def env_variables_for_docker_commands(self) -> dict[str, str]: _set_var(_env, "SKIP_SSH_SETUP", self.skip_ssh_setup) _set_var(_env, "SQLITE_URL", self.sqlite_url) _set_var(_env, "SSH_PORT", None, SSH_PORT) -_set_var(_env, "STANDALONE_DAG_PROCESSOR", self.standalone_dag_processor) +_set_var(_env, "STANDALONE_DAG_PROCESSOR", "true") Review Comment: Until we release 2.11 - yeah, we use breeze often (few people I know of) to test airflow releases and when they try to reproduce issues with older airflow versions. It's very convenient, because the old breeze version which is part of the 2.x release might or might not work for varios reasons related to dependency changes and other environmental factors, and would be great to keep it working for a while. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Always use standalone DAG processor in breeze [airflow]
jedcunningham commented on code in PR #45699: URL: https://github.com/apache/airflow/pull/45699#discussion_r1919182265 ## dev/breeze/src/airflow_breeze/params/shell_params.py: ## @@ -597,7 +596,7 @@ def env_variables_for_docker_commands(self) -> dict[str, str]: _set_var(_env, "SKIP_SSH_SETUP", self.skip_ssh_setup) _set_var(_env, "SQLITE_URL", self.sqlite_url) _set_var(_env, "SSH_PORT", None, SSH_PORT) -_set_var(_env, "STANDALONE_DAG_PROCESSOR", self.standalone_dag_processor) +_set_var(_env, "STANDALONE_DAG_PROCESSOR", "true") Review Comment: Hmm, do we need that flexibility? Seems reasonable that main could be 3.0+ only. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Always use standalone DAG processor in breeze [airflow]
potiuk commented on code in PR #45699: URL: https://github.com/apache/airflow/pull/45699#discussion_r1919165430 ## dev/breeze/src/airflow_breeze/params/shell_params.py: ## @@ -597,7 +596,7 @@ def env_variables_for_docker_commands(self) -> dict[str, str]: _set_var(_env, "SKIP_SSH_SETUP", self.skip_ssh_setup) _set_var(_env, "SQLITE_URL", self.sqlite_url) _set_var(_env, "SSH_PORT", None, SSH_PORT) -_set_var(_env, "STANDALONE_DAG_PROCESSOR", self.standalone_dag_processor) +_set_var(_env, "STANDALONE_DAG_PROCESSOR", "true") Review Comment: I think we need to make it conditional - when `--use-airflow-version` is set to < 2.10.* we probably don't want to run with standalone dag processor or at least give a chance to disable it. Possibly better solution will be to change `--use-standalone-dag-processor` to two way flag `--use-standalone-dag-processor/--no-use-standalone-dag-processor` and default it to `True` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Always use standalone DAG processor in breeze [airflow]
potiuk commented on code in PR #45699: URL: https://github.com/apache/airflow/pull/45699#discussion_r1919165430 ## dev/breeze/src/airflow_breeze/params/shell_params.py: ## @@ -597,7 +596,7 @@ def env_variables_for_docker_commands(self) -> dict[str, str]: _set_var(_env, "SKIP_SSH_SETUP", self.skip_ssh_setup) _set_var(_env, "SQLITE_URL", self.sqlite_url) _set_var(_env, "SSH_PORT", None, SSH_PORT) -_set_var(_env, "STANDALONE_DAG_PROCESSOR", self.standalone_dag_processor) +_set_var(_env, "STANDALONE_DAG_PROCESSOR", "true") Review Comment: I think we need to make it conditional - when `--use-airflow-version` is set to < 2.10.* we probably don't want to run with standalone dag processor or at least give a chance to disable it possibly better solution will be to change `--use-standalone-dag-processor` to two way flag `--use-standalone-dag-processor/--no-use-standalone-dag-processor` and default it to `True` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Always use standalone DAG processor in breeze [airflow]
potiuk commented on code in PR #45699:
URL: https://github.com/apache/airflow/pull/45699#discussion_r1919148438
##
Dockerfile.ci:
##
@@ -1053,7 +1053,7 @@ function start_webserver_with_examples(){
echo
echo "${COLOR_BLUE}Parsing example dags${COLOR_RESET}"
echo
-airflow scheduler --num-runs 100
+airflow dags reserialize
Review Comment:
Ah ... very nice!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
