Reamer commented on PR #4398:
URL: https://github.com/apache/zeppelin/pull/4398#issuecomment-1180412197
HI @zlosim
As you can see in the following lines, SPARK_SUBMIT_OPTIONS is concatenated
with the spark configurations specifically for Kubernetes.
https://github.com/apache/zeppelin/blob/8995ff227a44760b7dbb005a85bee4ac7772e1f2/zeppelin-plugins/launcher/k8s-standard/src/main/java/org/apache/zeppelin/interpreter/launcher/K8sRemoteInterpreterProcess.java#L323-L325
Your configuration values like `spark.driver.extraClassPath` and
`spark.driver.extraLibraryPath` should take no effect, because Zeppelin starts
Spark on Kubernetes always in client mode.
https://github.com/apache/zeppelin/blob/8995ff227a44760b7dbb005a85bee4ac7772e1f2/zeppelin-plugins/launcher/k8s-standard/src/main/java/org/apache/zeppelin/interpreter/launcher/K8sRemoteInterpreterProcess.java#L410
You can also set `SPARK_DRIVER_EXTRAJAVAOPTIONS_CONF` directly in your Spark
Zeppelin interpreter configuration if you want to override the (possibly empty)
default setting.
In general, I can recommend you to create a config map that contains all
Spark configurations and then include them in the containers via a custom
[interpreter-spec.yaml](https://github.com/apache/zeppelin/blob/master/k8s/interpreter/100-interpreter-spec.yaml)
in your Spark Zeppelin interpreter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
