villebro commented on PR #2462:
URL:
https://github.com/apache/datafusion-ballista/pull/2462#issuecomment-5718107295
@milenkovicm good point. Looking closer, I think there is an existing naming
mismatch here. Executor identity and executor routing are separate today:
executors autogenerate an opaque `executor_id` at process start, but the
scheduler calls them using `ExecutorMetadata { host, grpc_port, ... }`. For
schedulers, the value currently called `scheduler_id` is not equivalent to
`executor_id`. It is produced by `SchedulerConfig::scheduler_name()` as
`external_host:bind_port`, and executors use it directly as the callback
endpoint. I assume the reason for this has been to keep scheduler id more
tangible on logs etc.
To align with executor behavior, I think we should split these, so it would
be
- `scheduler_id` is an opaque id, like it is for executors.
- introduce `scheduler_host` + `scheduler_port`, or
`scheduler_grpc_endpoint` as a routable callback address used by executors.
..or something similar. That would make history entries a bit less
immediately tangible because they would carry an opaque scheduler id instead of
`host:port`, but I think it is probably the cleaner long-term model: it matches
executor semantics and avoids exposing internal routing details in history/API
state. Operators could still correlate via the scheduler startup log and
`/api/state`, which would expose the generated `scheduler_id`.
Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
