On Mon, Sep 8, 2014 at 10:00 AM, Dimension Data, LLC. < subscripti...@didata.us> wrote:
> user$ export MASTER=local[nn] # Run spark shell on LOCAL CPU threads. > user$ pyspark [someOptions] --driver-java-options -Dspark.*XYZ*.jar=' > /usr/lib/spark/assembly/lib/spark-assembly-*.jar' > > My question is, what to replace '*XYZ*' with in that case. > Ah, I see. There's no equivalent to that option in non-yarn mode, because either there is no need (e.g. in local mode everything is in the same machine) or the cluster backend doesn't support the feature (e.g. Spark Master does not have a distributed file cache like Yarn, at least as far as I know). That option is just to tell Yarn to use the distributed cache for the spark jar instead of auto-detecting where the jar is (which would incur in uploading the spark jar to all involved NMs). -- Marcelo
