andygrove opened a new issue, #6190: URL: https://github.com/apache/datafusion-comet/issues/6190
### Describe the bug The Spark Operator example in the Kubernetes guide ([kubernetes.md#L61-L100](https://github.com/apache/datafusion-comet/blob/67803a7a422c44de07af1e5d25c1dbeae8df68d4/docs/source/user-guide/latest/kubernetes.md?plain=1#L61-L100)) enables Comet through `spark.plugins`, but it does not set `spark.memory.offHeap.enabled` or `spark.memory.offHeap.size`. `CometDriverPlugin` therefore logs "Comet plugin is disabled because Spark is not running in off-heap mode", and the application runs without Comet. The example also sets no `spark.executor.memoryOverhead`, which the tuning guide says Comet needs. Kubernetes is where that matters most, because the pod limit is enforced. There are two more problems with the same example. It mixes versions: the image and the extra class path are Spark 3.5 with Scala 2.12, but `mainApplicationFile` is `spark-examples_2.13-4.1.3.jar` and `sparkVersion` is `4.1.3`. And `SparkPi` only uses RDDs, so it would not exercise Comet even with the plugin loaded. ### Expected behavior The example sets off-heap memory and the memory overhead, uses consistent versions, and runs a workload that Comet accelerates. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
