Hello folks, I have a spark app with a custom implementation of *fs.s3a.s3.client.factory.impl* which is packaged into the same jar. Output of *jar tf*
*2620 Mon Dec 12 11:23:00 IST 2022 aws/utils/MyS3ClientFactory.class* However when I run the my spark app with spark-submit in cluster mode, it fails with the following error: *java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class aws.utils.MyS3ClientFactory not found* I tried: 1. passing in the jar to the *--jars* option (with the local path) 2. Passing in the jar to *spark.yarn.jars* option with an HDFS path but still the same error. Any suggestions on what I'm missing? Other pertinent details: Spark version: 3.3.0 Hadoop version: 3.3.4 Command used to run the app */spark/bin/spark-submit --class MyMainClass --deploy-mode cluster --master yarn --conf spark.executor.instances=6 /path/to/my/jar* TIA!
