[PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-03-26 Thread via GitHub
o-nikolas opened a new pull request, #38514: URL: https://github.com/apache/airflow/pull/38514 This PR delivers the first set of changes to the scheduler job to transform it to support hybrid executors. It covers starting and ending the executors as well as debug dump.

Re: [PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-04-01 Thread via GitHub
o-nikolas commented on code in PR #38514: URL: https://github.com/apache/airflow/pull/38514#discussion_r1540104792 ## airflow/jobs/job.py: ## @@ -104,12 +105,13 @@ class Job(Base, LoggingMixin): Only makes sense for SchedulerJob and BackfillJob instances. """ -de

Re: [PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-04-01 Thread via GitHub
uranusjr commented on code in PR #38514: URL: https://github.com/apache/airflow/pull/38514#discussion_r1547079296 ## airflow/jobs/scheduler_job_runner.py: ## @@ -819,19 +821,20 @@ def _execute(self) -> int | None: ) try: -self.job.executor.job

Re: [PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-04-02 Thread via GitHub
o-nikolas commented on code in PR #38514: URL: https://github.com/apache/airflow/pull/38514#discussion_r1548204315 ## airflow/jobs/scheduler_job_runner.py: ## @@ -819,19 +821,20 @@ def _execute(self) -> int | None: ) try: -self.job.executor.jo

Re: [PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-04-02 Thread via GitHub
dstandish commented on code in PR #38514: URL: https://github.com/apache/airflow/pull/38514#discussion_r1548637266 ## airflow/jobs/job.py: ## @@ -104,12 +105,13 @@ class Job(Base, LoggingMixin): Only makes sense for SchedulerJob and BackfillJob instances. """ -de

Re: [PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-04-02 Thread via GitHub
dstandish commented on code in PR #38514: URL: https://github.com/apache/airflow/pull/38514#discussion_r1548638498 ## airflow/jobs/scheduler_job_runner.py: ## @@ -270,8 +270,10 @@ def _debug_dump(self, signum: int, frame: FrameType | None) -> None: self.log.info("%s\

Re: [PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-04-02 Thread via GitHub
dstandish commented on code in PR #38514: URL: https://github.com/apache/airflow/pull/38514#discussion_r1548638986 ## airflow/jobs/scheduler_job_runner.py: ## @@ -270,8 +270,10 @@ def _debug_dump(self, signum: int, frame: FrameType | None) -> None: self.log.info("%s\

Re: [PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-04-02 Thread via GitHub
dstandish commented on code in PR #38514: URL: https://github.com/apache/airflow/pull/38514#discussion_r1548639831 ## airflow/jobs/scheduler_job_runner.py: ## @@ -819,19 +821,20 @@ def _execute(self) -> int | None: ) try: -self.job.executor.jo

Re: [PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-04-02 Thread via GitHub
o-nikolas commented on code in PR #38514: URL: https://github.com/apache/airflow/pull/38514#discussion_r1548690491 ## airflow/jobs/job.py: ## @@ -104,12 +105,13 @@ class Job(Base, LoggingMixin): Only makes sense for SchedulerJob and BackfillJob instances. """ -de

Re: [PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-04-02 Thread via GitHub
o-nikolas commented on code in PR #38514: URL: https://github.com/apache/airflow/pull/38514#discussion_r1548693252 ## airflow/jobs/scheduler_job_runner.py: ## @@ -270,8 +270,10 @@ def _debug_dump(self, signum: int, frame: FrameType | None) -> None: self.log.info("%s\

Re: [PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-04-02 Thread via GitHub
o-nikolas commented on code in PR #38514: URL: https://github.com/apache/airflow/pull/38514#discussion_r1548690491 ## airflow/jobs/job.py: ## @@ -104,12 +105,13 @@ class Job(Base, LoggingMixin): Only makes sense for SchedulerJob and BackfillJob instances. """ -de

Re: [PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-04-02 Thread via GitHub
o-nikolas commented on PR #38514: URL: https://github.com/apache/airflow/pull/38514#issuecomment-2033219581 > on executor end... i looked at the way executors have implemented end... i was surprised to see that they wait for all tasks to finish. given that we have re-adoption logic, i wonde

Re: [PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-04-02 Thread via GitHub
o-nikolas commented on code in PR #38514: URL: https://github.com/apache/airflow/pull/38514#discussion_r1548704690 ## airflow/jobs/scheduler_job_runner.py: ## @@ -819,19 +821,20 @@ def _execute(self) -> int | None: ) try: -self.job.executor.jo

Re: [PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-04-02 Thread via GitHub
uranusjr commented on code in PR #38514: URL: https://github.com/apache/airflow/pull/38514#discussion_r1548745509 ## airflow/jobs/job.py: ## @@ -104,12 +105,13 @@ class Job(Base, LoggingMixin): Only makes sense for SchedulerJob and BackfillJob instances. """ -def

Re: [PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-04-03 Thread via GitHub
dstandish commented on code in PR #38514: URL: https://github.com/apache/airflow/pull/38514#discussion_r1549952673 ## airflow/jobs/job.py: ## @@ -104,12 +105,13 @@ class Job(Base, LoggingMixin): Only makes sense for SchedulerJob and BackfillJob instances. """ -de

Re: [PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-04-03 Thread via GitHub
dstandish commented on code in PR #38514: URL: https://github.com/apache/airflow/pull/38514#discussion_r1550099456 ## airflow/jobs/scheduler_job_runner.py: ## @@ -270,8 +270,10 @@ def _debug_dump(self, signum: int, frame: FrameType | None) -> None: self.log.info("%s\

Re: [PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-04-04 Thread via GitHub
o-nikolas commented on code in PR #38514: URL: https://github.com/apache/airflow/pull/38514#discussion_r1552549164 ## airflow/jobs/scheduler_job_runner.py: ## @@ -270,8 +270,10 @@ def _debug_dump(self, signum: int, frame: FrameType | None) -> None: self.log.info("%s\

Re: [PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-04-04 Thread via GitHub
dstandish commented on code in PR #38514: URL: https://github.com/apache/airflow/pull/38514#discussion_r1552594898 ## airflow/jobs/scheduler_job_runner.py: ## @@ -270,8 +270,10 @@ def _debug_dump(self, signum: int, frame: FrameType | None) -> None: self.log.info("%s\

Re: [PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-04-05 Thread via GitHub
o-nikolas commented on code in PR #38514: URL: https://github.com/apache/airflow/pull/38514#discussion_r1554018702 ## airflow/jobs/scheduler_job_runner.py: ## @@ -819,19 +821,20 @@ def _execute(self) -> int | None: ) try: -self.job.executor.jo

Re: [PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-04-07 Thread via GitHub
potiuk commented on code in PR #38514: URL: https://github.com/apache/airflow/pull/38514#discussion_r1555067360 ## airflow/jobs/job.py: ## @@ -104,12 +105,13 @@ class Job(Base, LoggingMixin): Only makes sense for SchedulerJob and BackfillJob instances. """ -def _

Re: [PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-04-07 Thread via GitHub
uranusjr commented on code in PR #38514: URL: https://github.com/apache/airflow/pull/38514#discussion_r1555276527 ## airflow/jobs/job.py: ## @@ -104,12 +106,13 @@ class Job(Base, LoggingMixin): Only makes sense for SchedulerJob and BackfillJob instances. """ -def

Re: [PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-04-08 Thread via GitHub
o-nikolas commented on code in PR #38514: URL: https://github.com/apache/airflow/pull/38514#discussion_r1556103085 ## airflow/jobs/job.py: ## @@ -104,12 +106,13 @@ class Job(Base, LoggingMixin): Only makes sense for SchedulerJob and BackfillJob instances. """ -de

Re: [PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-04-09 Thread via GitHub
o-nikolas merged PR #38514: URL: https://github.com/apache/airflow/pull/38514 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflo