Re: foreachPartition's operation is taking long to finish

On Sun, Dec 18, 2016 at 2:26 AM, vaquar khan wrote: > select * from indexInfo; > Hi Vaquar I do not see CF with the name indexInfo in any of the cassandra databases. Thank Deepak -- Thanks Deepak www.bigdatabig.com www.keosha.net

Re: foreachPartition's operation is taking long to finish

There are 8 worker nodes in the cluster . Thanks Deepak On Dec 18, 2016 2:15 AM, "Holden Karau" wrote: > How many workers are in the cluster? > > On Sat, Dec 17, 2016 at 12:23 PM Deepak Sharma > wrote: > >> Hi All, >> I am iterating over data

Re: foreachPartition's operation is taking long to finish

Hi Deepak, Could you share Index information in your database. select * from indexInfo; Regards, Vaquar khan On Sat, Dec 17, 2016 at 2:45 PM, Holden Karau wrote: > How many workers are in the cluster? > > On Sat, Dec 17, 2016 at 12:23 PM Deepak Sharma

Re: foreachPartition's operation is taking long to finish

How many workers are in the cluster? On Sat, Dec 17, 2016 at 12:23 PM Deepak Sharma wrote: > Hi All, > I am iterating over data frame's paritions using df.foreachPartition . > Upon each iteration of row , i am initializing DAO to insert the row into > cassandra. > Each of

foreachPartition's operation is taking long to finish

Hi All, I am iterating over data frame's paritions using df.foreachPartition . Upon each iteration of row , i am initializing DAO to insert the row into cassandra. Each of these iteration takes almost 1 and half minute to finish. In my workflow , this is part of an action and 100 partitions are