I successfully submitted and ran org.apache.spark.examples.SparkPi on Yarn using 4.0.0-preview1. However I got it to work only after fixing an issue with the Yarn nodemanagers (Hadoop v3.3.6 and v3.4.0). Namely the issue was: 1. If the nodemanagers used java 11, Yarn threw an error about not finding the jdk.incubator.vector module. 2. If the nodemanagers used java 17, which has the jdk.incubator.vector module, Yarn threw a reflection error about class not found.
To resolve the error and successfully calculate pi, 1. I ran java 17 on the nodemanagers and 2. added 'export HADOOP_OPTS="--add-opens=java.base/java.lang=ALL-UNNAMED"' to their conf/hadoop-env.sh file. George
