There is a design document that covers a lot of concerns:
https://docs.google.com/document/d/1pcyH5f610X2jyJW9WbWHnj8jktQPLlbbmmUwdeK4fJk,
validation included.
We had a discussion about validation (validate before we hit the api
server) and was considered too much. In general regarding Rob's
I can speak somewhat to the current design. Two of the goals for the design
of this feature are that
(1) its behavior is easy to reason about
(2) its implementation in the back-end is light weight
Option 1 was chosen partly because it's behavior is relatively simple to
describe to a user: "Your
Thanks for bring this up. My opinion on this is this feature is really
targeting advanced use cases that need more customization than what the
basic k8s-related Spark config properties offer. So I think it's fair to
assume that users who would like to use this feature know the risks and are
Hey all
For those following the K8S backend you are probably aware of SPARK-24434 [1]
(and PR 22416 [2]) which proposes a mechanism to allow for advanced pod
customisation via pod templates. This is motivated by the fact that
introducing additional Spark configuration properties for each
