dongjoon-hyun commented on PR #58190: URL: https://github.com/apache/spark/pull/58190#issuecomment-5458049374
One more thing about the executor side, which the updated description now mentions. `BasicExecutorFeatureStep` sets `ENV_SPARK_USER` from `Utils.getCurrentUserName()` as well, and that call is evaluated inside the driver JVM, so it picks up whatever `SPARK_USER` this patch put into the driver container. Executor pods therefore inherit the proxy user too. That looks like the right behavior to me (driver and executors should agree on the identity), but it is worth being explicit that it is intended rather than incidental. Note that it is not straightforward to cover with a unit test: `proxyUser` lives on `KubernetesDriverConf` only, so `BasicExecutorFeatureStep` cannot read it directly, and the propagation goes through the process environment, which `BasicExecutorFeatureStepSuite` cannot manipulate. So I am not asking for a new unit test here. Instead, could you confirm you verified this end to end on a real cluster -- i.e. submit with `--proxy-user alice` and check that both the driver and the executor pods show `SPARK_USER=alice`, and that the driver's Hadoop operations run as `alice`? A short note in the "How was this patch tested?" section would be enough. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
