I am using CDH5.1 and Spark 1.0.0.
Trying to configure resources to be allocated to each application. How do I
do this? For example, I would each app to use 2 cores and 8G of RAM. I have
tried using the pyspark commandline paramaters for --driver-memory,
--driver-cores and see no effect of those
Hello,
What do you mean by app that uses 2 cores and 8G of RAM?
Spark apps generally involve multiple processes. The command line
options you used affect only one of them (the driver). You may want to
take a look at similar configuration for executors. Also, check the
documentation:
