I am splitting an integer array in 2 partitions and using an accumulator to sum the array. problem is
1. I am not seeing execution time becoming half of a linear summing. 2. The second node (from looking at timestamps) takes 3 times as long as the first node. This gives the impression it is "waiting" for the first node to finish. Hence, I am given the impression using accumulator.sum () in the kernel and rdd.foreach (kernel) is making things sequential. Any api/setting suggestions where I could make things parallel ? On Mon, May 9, 2016 at 8:24 PM, Abi <analyst.tech.j...@gmail.com> wrote: > I am splitting an integer array in 2 partitions and using an accumulator > to sum the array. problem is > > 1. I am not seeing execution time becoming half of a linear summing. > > 2. The second node (from looking at timestamps) takes 3 times as long as > the first node. This gives the impression it is "waiting" for the first > node to finish. > > Hence, I am given the impression using accumulator.sum () in the kernel > and rdd.foreach (kernel) is making things sequential. > > Any api/setting suggestions where I could make things parallel ? > > >
