Re: [PR] Remove option for unlimited parallelism [airflow]
jscheffl merged PR #48218: URL: https://github.com/apache/airflow/pull/48218 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Remove option for unlimited parallelism [airflow]
jscheffl commented on PR #48218: URL: https://github.com/apache/airflow/pull/48218#issuecomment-2759541390 No objections received. Merging. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Remove option for unlimited parallelism [airflow]
jscheffl commented on code in PR #48218: URL: https://github.com/apache/airflow/pull/48218#discussion_r2015079539 ## airflow-core/newsfragments/48218.significant.rst: ## @@ -0,0 +1,14 @@ +Remove option for unlimited parallelism. + +Before Airflow 3.0 it was possible to set unlimited parallelism by setting ``[core] parallelism`` to 0. This was removed in Airflow 3.0.0. + +* Types of change + + * [ ] Dag changes + * [x] Config changes + * [ ] API changes + * [ ] CLI changes + * [x] Behaviour changes + * [ ] Plugin changes + * [ ] Dependency changes + * [ ] Code interface changes Review Comment: Here we are :-) #48423 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Remove option for unlimited parallelism [airflow]
jscheffl commented on code in PR #48218: URL: https://github.com/apache/airflow/pull/48218#discussion_r2014914644 ## airflow-core/newsfragments/48218.significant.rst: ## @@ -0,0 +1,14 @@ +Remove option for unlimited parallelism. + +Before Airflow 3.0 it was possible to set unlimited parallelism by setting ``[core] parallelism`` to 0. This was removed in Airflow 3.0.0. + +* Types of change + + * [ ] Dag changes + * [x] Config changes + * [ ] API changes + * [ ] CLI changes + * [x] Behaviour changes + * [ ] Plugin changes + * [ ] Dependency changes + * [ ] Code interface changes Review Comment: Yeah, I hope I can make the config lint addition in a separate PR the next days. But if you are faster than me, this is welcome. I assume the value should not be auto-migrated (to what?) but an error should be raised. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Remove option for unlimited parallelism [airflow]
Lee-W commented on code in PR #48218: URL: https://github.com/apache/airflow/pull/48218#discussion_r2013637255 ## airflow-core/newsfragments/48218.significant.rst: ## @@ -0,0 +1,14 @@ +Remove option for unlimited parallelism. + +Before Airflow 3.0 it was possible to set unlimited parallelism by setting ``[core] parallelism`` to 0. This was removed in Airflow 3.0.0. + +* Types of change + + * [ ] Dag changes + * [x] Config changes + * [ ] API changes + * [ ] CLI changes + * [x] Behaviour changes + * [ ] Plugin changes + * [ ] Dependency changes + * [ ] Code interface changes Review Comment: Do we want to check whether `core.parallelism` is 0? if so, we'll need to add a migration rule in `airflow config lint`. You can leave the implementation part to @sunank200 and me but if we want to add it. We'll need to list it down in the newsfragments ## airflow-core/newsfragments/48218.significant.rst: ## @@ -0,0 +1,14 @@ +Remove option for unlimited parallelism. + +Before Airflow 3.0 it was possible to set unlimited parallelism by setting ``[core] parallelism`` to 0. This was removed in Airflow 3.0.0. + +* Types of change + + * [ ] Dag changes + * [x] Config changes + * [ ] API changes + * [ ] CLI changes + * [x] Behaviour changes + * [ ] Plugin changes + * [ ] Dependency changes + * [ ] Code interface changes Review Comment: Do we want to check whether `core.parallelism` is 0? if so, we'll need to add a migration rule in `airflow config lint`. You can leave the implementation part to @sunank200 and me but if we want to add it. We'll need to list it down in the newsfragments 🙂 Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Remove option for unlimited parallelism [airflow]
jscheffl commented on PR #48218: URL: https://github.com/apache/airflow/pull/48218#issuecomment-2752585630 > Given the discussion in #41055 I think we better have mailing list discussion on this. WDYT? I'll drop a note on devlist. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Remove option for unlimited parallelism [airflow]
jedcunningham commented on code in PR #48218: URL: https://github.com/apache/airflow/pull/48218#discussion_r2013007248 ## newsfragments/48218.significant.rst: ## Review Comment: It's in [here](https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/cli/commands/config_command.py). But we don't really have a "this value isn't okay any more" ability, which we could add... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Remove option for unlimited parallelism [airflow]
jscheffl commented on code in PR #48218: URL: https://github.com/apache/airflow/pull/48218#discussion_r2012963142 ## newsfragments/48218.significant.rst: ## Review Comment: @jedcunningham took a monment until it also came into my mind... to we need to add it to "airflow config lint"? Don't know where but if you can enlighten me I am okay :-D -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Remove option for unlimited parallelism [airflow]
jedcunningham commented on code in PR #48218: URL: https://github.com/apache/airflow/pull/48218#discussion_r2012193047 ## newsfragments/48218.significant.rst: ## Review Comment: Sorry, for some reason I thought we also had config rules in ruff, but looks like we don't. Sorry for the noise. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Remove option for unlimited parallelism [airflow]
eladkal commented on PR #48218: URL: https://github.com/apache/airflow/pull/48218#issuecomment-2750206423 Given the discussion in #41055 I think we better have mailing list discussion on this. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Remove option for unlimited parallelism [airflow]
jscheffl commented on code in PR #48218: URL: https://github.com/apache/airflow/pull/48218#discussion_r2011366692 ## newsfragments/48218.significant.rst: ## Review Comment: It is a standard configuration in the deployment, nothing that is needed to change in DAGs. Therefore I assume a ruff rule does not help really? I could add one but I assume an auto-fix is anyway not possible and if you had it to 0 previously it will fail staring. I would think that makes it clear. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Remove option for unlimited parallelism [airflow]
jedcunningham commented on code in PR #48218: URL: https://github.com/apache/airflow/pull/48218#discussion_r2011168342 ## newsfragments/48218.significant.rst: ## Review Comment: We should probably get a ruff rule for this as well, could you add a migration rules section to this so those folks can make sure that gets done? ## newsfragments/48218.significant.rst: ## @@ -0,0 +1,14 @@ +Remove option for unlimited parallelism. + +Before Airflow 3.0 it was possible to set unlimited parallelism by setting it to 0. This was removed in Airflow 3.0.0. Review Comment: ```suggestion Before Airflow 3.0 it was possible to set unlimited parallelism by setting `[core] parallelism` to 0. This was removed in Airflow 3.0.0. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
[PR] Remove option for unlimited parallelism [airflow]
jscheffl opened a new pull request, #48218: URL: https://github.com/apache/airflow/pull/48218 Last moment as I was looking through the open tasks to make some cleanup before Airflow 3.0 closes: #41162 related: #41055, #41107 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
