Re: Spark-Connect: Param `--packages` does not take effect for executors.

The issue you're encountering with the iceberg-spark-runtime dependency not being properly passed to the executors in your Spark Connect server deployment could be due to a couple of factors: 1. *Spark Submit Packaging:* When you use the --packages parameter in spark-submit, it only

Re: Spark / Scala conflict

The error message Caused by: java.lang.ClassNotFoundException: scala.Product$class indicates that the Spark job is trying to load a class that is not available in the classpath. This can happen if the Spark job is compiled with a different version of Scala than the version of Scala that is used to

Re: JDK version support information

Spark version 2.4.5.1 is based on Apache Spark 2.4.5. According to the official Spark documentation for version 2.4.5, the maximum supported JDK (Java Development Kit) version is JDK 8 (Java 8). Spark 2.4.5 is not compatible with JDK versions higher than Java 8. Therefore, you should use JDK 8 to

Re:

Generally it is never a good idea to run processes as root on any production machines. The main problem is the security problems not found or disclosed, so if someone malicious takes advantage of a vulnerability like the ones described below, they can first get in, and little by little escalate

unsubscribe

-- Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman

Re: shuffle mathematic formulat

I would have to check it, but in principle it could be done by checking the streaming logs, so that once you detect when a shuffle operation starts and ends, you can know the total operation time. https://stackoverflow.com/questions/27276884/what-is-shuffle-read-shuffle-write-in-apache-spark El