How many workers are in the cluster? On Sat, Dec 17, 2016 at 12:23 PM Deepak Sharma <deepakmc...@gmail.com> wrote:
> Hi All, > I am iterating over data frame's paritions using df.foreachPartition . > Upon each iteration of row , i am initializing DAO to insert the row into > cassandra. > Each of these iteration takes almost 1 and half minute to finish. > In my workflow , this is part of an action and 100 partitions are being > created for the df as i can see 100 tasks being created , where the insert > dao operation is being performed. > Since each of these 100 tasks , takes around 1 and half minute to complete > , it takes around 2 hour for this small insert operation. > Is anyone facing the same scenario and is there any time efficient way to > handle this? > This latency is not good in out use case. > Any pointer to improve/minimise the latency will be really appreciated. > > > -- > Thanks > Deepak > > >
