milenkovicm commented on code in PR #2813: URL: https://github.com/apache/datafusion-ballista/pull/2813#discussion_r4516093540
########## docs/source/upgrading/56.0.0.md: ########## @@ -196,6 +196,28 @@ cannot re-run another job's output, so a caller must fail them (the scheduler posts `JobRunningFailed`). A caller that only needs the tasks to cancel can take the first element. +#### Partition locations hold an `ExecutorEndpoint` + +A partition location only has to say where to fetch from, so +`PartitionLocation::executor_meta` and +`PipelinedPartitionLocation::executor_meta` are now `Arc<ExecutorEndpoint>`, a +new type with the executor's `id`, `host` and `port`, shared between locations +rather than copied into each one. `ExecutorMetadata` is unchanged, and is still +what the scheduler records about a registered executor. To build an endpoint +from one, use `Arc::new(ExecutorEndpoint::from(&metadata))`. Code that read +`grpc_port`, `specification` or `os_info` from a location should look up the +`ExecutorMetadata` by `id` instead. + +Shuffle reader plan nodes now list each executor once and their locations refer +to it by position. This changes the wire format between schedulers and +executors, so `BALLISTA_PROTOCOL_VERSION` is now `4` and they must be upgraded +together. + +In the protobuf definitions, the `ExecutorMetadata` message is renamed +`ExecutorEndpoint` and loses `grpc_port`, `specification` and `os_info`. Job +statuses still set `executor_meta` on every location, so clients that read its +`id`, `host` or `port` do not need to change. Review Comment: do we need this paragraph? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
