yalindogusahin commented on PR #794: URL: https://github.com/apache/spark-kubernetes-operator/pull/794#issuecomment-5568010138
Thanks for the detailed review. I went through each point and pushed the fixes. **Env-var collision** The docs and example now use `SPARK_DAEMON_JAVA_OPTS` instead of `SPARK_WORKER_OPTS`. The operator appends its own (sometimes empty) `SPARK_WORKER_OPTS` to the worker container after merging the pod template, which silently overwrote the user value. `SPARK_DAEMON_JAVA_OPTS` is read by Spark`s `SparkClassCommandBuilder` for the Worker and the operator never sets it, so nothing collides. **API shape** Both fields now sit under `workerSpec.networkPolicy.metricsPort` and `workerSpec.networkPolicy.metricsIngress`, matching `operatorDeployment.networkPolicy.metricsIngress`. It also clears up the naming confusion; `networkPolicy` now reads as a policy block rather than two loose metrics fields. `metricsPort: 8081` (the web UI port) is now rejected with an `IllegalArgumentException` and a unit test, since the UI and its metrics endpoint share the same embedded HTTP server. **Docs / example** Dropped the "complete, runnable" wording and put a note about the custom-image requirement next to the image field. Removed the unrelated `spark.ui.reverseProxy`. I kept the `jmx_prometheus_javaagent` version in the Dockerfile `ADD` URL because that needs a concrete artifact. Happy to remove it if you would rather not pin it. **Housekeeping** Rebased onto the latest main, and added a `tests/e2e/worker-metrics-network-policy/` Chainsaw suite that checks the generated worker NetworkPolicy. Local build is green: `:spark-operator-api:build`, `:spark-submission-worker:build`, `javadoc`, and `assertGeneratedCRDMatchesHelmChart` all pass, and the CRD is in sync with the Helm chart. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
