Hi, I would refer to their documentation to better understand the concepts behind cluster overview and submitting applications:
- https://spark.apache.org/docs/latest/cluster-overview.html#cluster-manager-types - https://spark.apache.org/docs/latest/submitting-applications.html When using local[*] you can get as many worker threads as your cores in the same jvm running your driver and not executors. If you want to test against a real cluster you can look into using stand-alone mode. HTH, Riccardo On Mon, Oct 23, 2023 at 5:31 PM KhajaAsmath Mohammed < mdkhajaasm...@gmail.com> wrote: > Hi, > > I am running a spark job in spark EC2 machine whiich has 40 cores. Driver > and executor memory is 16 GB. I am using local[*] but I still get only one > executor(driver). Is there a way to get more executors with this config. > > I am not using yarn or mesos in this case. Only one machine which is > enough for our work load but the data is increased. > > Thanks, > Asmath >
