I see the motivation, but does it have to look so bad?
The subclass will look like this:
class SchedulerJobRunnerLinearTIScan(SchedulerJobRunner):
def __init__(
self,
job: Job,
num_runs: int = conf.getint("scheduler", "num_runs"),
scheduler_idle_sleep_time: float = conf.getfloat("scheduler",
"scheduler_idle_sleep_time"),
log: Logger | None = None,
):
super().__init__(
job=job,
num_runs=num_runs,
scheduler_idle_sleep_time=scheduler_idle_sleep_time,
log=log,
)
self.task_selector = TASK_SELECTORS[LINEAR_SCAN_SELECTOR]
The super class will use the injected hard-coded task selector.
Can't we introduce a configuration hierarchy like `core.internal` and put there
things *not* exposed to the end user? So we don't have to do this weird
subclassing?
It will look thus:
class SchedulerJobRunner(...):
task_selector_type = conf.get("scheduler.internal",
"task_selector_strategy")
self.task_selector = TASK_SELECTORS[task_selector_type]
We'd just like to have an internal toggle as an implementation detail, which
won't be undocumented and custom implementations *won't* be supported. It's
just more convenient and straightforward.
Mb there's another way of internal settings management I missed?
On Tuesday, September 16th, 2025 at 11:34 AM, Ash Berlin-Taylor
<[email protected]> wrote:
> > On 16 Sep 2025, at 08:58, asquator [email protected] wrote:
> >
> > Yes, exposing pluggable features means fixing an API, which is confining
> > and just hard to do given the current implementation
>
>
>
> class MyScheduler:
> def execute(self):
> while True:
> # Do what ever you want.
>
> `airflow scheduler --impl=my.module.MyScheduler`
>
> That is the API.
>
> That is as pluggable as we need it to be.
>
> Everything can be built on top of that, including if you want it, a pluggable
> task selection mechanisms.
>
> Airflow already has too many config options and ways of tuning behaviour. We
> need less of them, not more.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
