Data is not distributed to executors by anything. If you are processing data with Spark. Spark spawns tasks on executors to read chunks of data from wherever they are (S3, HDFS, etc).
On Mon, Jun 6, 2022 at 4:07 PM Sid <flinkbyhe...@gmail.com> wrote: > Hi experts, > > > When we load any file, I know that based on the information in the spark > session about the executors location, status and etc , the data is > distributed among the worker nodes and executors. > > But I have one doubt. Is the data initially loaded on the driver and then > it is distributed or it is directly distributed amongst the workers? > > Thanks, > Sid >
