I am facing a general problem actually, which seem to be related to how
many JVM get launched.
In my map task I read a file and fill a map out of it.
Now, since the data is static and map tasks are called for every record of
RDD and I want to read it only once, so I kept the map as static (in Java)
ya ya had got that. Thx.
On Sun, Jan 5, 2014 at 1:41 AM, Roshan Nair wrote:
> The driver jvm is the jvm in which you create the sparkContext and launch
> your job. Its different from the master and worker daemons.
>
> Roshan
> On Jan 5, 2014 1:37 AM, "Archit Thakur" wrote:
>
>> Yeah, I believe
The driver jvm is the jvm in which you create the sparkContext and launch
your job. Its different from the master and worker daemons.
Roshan
On Jan 5, 2014 1:37 AM, "Archit Thakur" wrote:
> Yeah, I believed that too.
>
> The last being the jvm in which your driver runs.??? Isn't it in the 3
> wo
Oh, you meant main driver. Yes, correct.
On Sun, Jan 5, 2014 at 1:36 AM, Archit Thakur wrote:
> Yeah, I believed that too.
>
> The last being the jvm in which your driver runs.??? Isn't it in the 3
> worker daemon, we have already considered.
>
>
> On Sun, Jan 5, 2014 at 1:28 AM, Roshan Nair wr
Yeah, I believed that too.
The last being the jvm in which your driver runs.??? Isn't it in the 3
worker daemon, we have already considered.
On Sun, Jan 5, 2014 at 1:28 AM, Roshan Nair wrote:
> I missed this. Its actually 1+3+3+1. The last being the jvm in which your
> driver runs.
>
> Roshan
I missed this. Its actually 1+3+3+1. The last being the jvm in which your
driver runs.
Roshan
On Jan 5, 2014 1:24 AM, "Roshan Nair" wrote:
> Hi Archit,
>
> I believe its the last case - 1+3+3.
>
> From what I've seen its one jvm per worker per spark application.
>
> You will have multiple thread
Hi Archit,
I believe its the last case - 1+3+3.
>From what I've seen its one jvm per worker per spark application.
You will have multiple threads within a worker jvm working on different
partitions concurrently. The number of partitions that a worker handles
concurrently appears to be determined
A JVM reuse doubt.
Lets say I have a job which has 5 stages:
Each stage has 10 tasks(10 partitions) Each task has 3 transformation.
My Cluster is size 4 (1 Master, 3 Workers), How many JVMs will be launched?
1-Master Daemon 3-Worker Daemon
JVM = 1+3+10*3*5 (where at a time 10 will be executed para
