Hey, I love when questions are numbered, it's easier :)
1) Yes (but I am not an expert) 2) You don't control... One of my process is going to 8k tasks, so... 3) Yes, if you have HT, it double. My servers have 12 cores, but HT, so it makes 24. 4) From my understanding: Slave is the logical computational unit and Worker is really the one doing the job. 5) Dunnoh 6) Dunnoh > On Jul 20, 2016, at 1:30 PM, Sachin Mittal <sjmit...@gmail.com> wrote: > > Hi, > I was able to build and run my spark application via spark submit. > > I have understood some of the concepts by going through the resources at > https://spark.apache.org <https://spark.apache.org/> but few doubts still > remain. I have few specific questions and would be glad if someone could > share some light on it. > > So I submitted the application using spark.master local[*] and I have a 8 > core PC. > > - What I understand is that application is called as job. Since mine had two > stages it gets divided into 2 stages and each stage had number of tasks which > ran in parallel. > Is this understanding correct. > > - What I notice is that each stage is further divided into 262 tasks From > where did this number 262 came from. Is this configurable. Would increasing > this number improve performance. > > - Also I see that the tasks are run in parallel in set of 8. Is this because > I have a 8 core PC. > > - What is the difference or relation between slave and worker. When I did > spark-submit did it start 8 slaves or worker threads? > > - I see all worker threads running in one single JVM. Is this because I did > not start slaves separately and connect it to a single master cluster > manager. If I had done that then each worker would have run in its own JVM. > > - What is the relationship between worker and executor. Can a worker have > more than one executors? If yes then how do we configure that. Does all > executor run in the worker JVM and are independent threads. > > I suppose that is all for now. Would appreciate any response.Will add > followup questions if any. > > Thanks > Sachin > >
