Hi Spark users/experts,
I’m wondering how does the Spark scheduler work?
What kind of resources will be considered during the scheduling, does it
include the disk resources or I/O resources, e.g., number of IO ports.
Is network resources considered in that?
My understanding is that only CPU
-scheduling.html#scheduling-within-an-application>
>
> On Mon, Apr 11, 2016 at 3:15 PM, Jialin Liu <jaln...@lbl.gov
> <mailto:jaln...@lbl.gov>> wrote:
> Hi Spark users/experts,
>
> I’m wondering how does the Spark scheduler work?
> What kind of resources will b
Hi,
I have set the partitions as 6000, and requested 100 nodes, with 32
cores each node,
and the number of executors is 32 per node
spark-submit --master $SPARKURL --executor-cores 32 --driver-memory
20G --executor-memory 80G single-file-test.py
And I'm reading a 2.2 TB, the code, just has
