Re: spark 1.6 foreachPartition only appears to be running on one executor

2016-03-11 Thread Darin McBeath
:57 PM Subject: Re: spark 1.6 foreachPartition only appears to be running on one executor My driver code has the following: // Init S3 (workers) so we can read the assets partKeyFileRDD.foreachPartition(new SimpleStorageServiceInit(arg1, arg2, arg3)); // Get the assets. Create a key pair where t

Re: spark 1.6 foreachPartition only appears to be running on one executor

2016-03-11 Thread Darin McBeath
_ From: Jacek Laskowski <ja...@japila.pl> To: Darin McBeath <ddmcbe...@yahoo.com> Cc: user <user@spark.apache.org> Sent: Friday, March 11, 2016 1:40 PM Subject: Re: spark 1.6 foreachPartition only appears to be running on one executor Hi, Could you share the code

Re: spark 1.6 foreachPartition only appears to be running on one executor

2016-03-11 Thread Jacek Laskowski
k Laskowski <ja...@japila.pl> > To: Darin McBeath <ddmcbe...@yahoo.com> > Cc: user <user@spark.apache.org> > Sent: Friday, March 11, 2016 1:24 PM > Subject: Re: spark 1.6 foreachPartition only appears to be running on one > executor > > > > Hi, > How do

Re: spark 1.6 foreachPartition only appears to be running on one executor

2016-03-11 Thread Darin McBeath
dies. Darin. From: Jacek Laskowski <ja...@japila.pl> To: Darin McBeath <ddmcbe...@yahoo.com> Cc: user <user@spark.apache.org> Sent: Friday, March 11, 2016 1:24 PM Subject: Re: spark 1.6 foreachPartition only appears to be running on one executor

Re: spark 1.6 foreachPartition only appears to be running on one executor

2016-03-11 Thread Jacek Laskowski
Hi, How do you check which executor is used? Can you include a screenshot of the master's webUI with workers? Jacek 11.03.2016 6:57 PM "Darin McBeath" napisał(a): > I've run into a situation where it would appear that foreachPartition is > only running on one of my

spark 1.6 foreachPartition only appears to be running on one executor

2016-03-11 Thread Darin McBeath
I've run into a situation where it would appear that foreachPartition is only running on one of my executors. I have a small cluster (2 executors with 8 cores each). When I run a job with a small file (with 16 partitions) I can see that the 16 partitions are initialized but they all appear to