Networking Exceptions in Spark 1.6.1 with Dynamic Allocation and YARN Pre-Emption

Hey all, We're working on setting up a Spark 1.6.1 cluster on Amazon EC2, and encountering some problems related to pre-emption. We have followed all the instructions for setting up dynamic allocation, including enabling the external spark shuffle service in the YARN NodeManagers. When a

Re: Trying to understand dynamic resource allocation

My understanding is that dynamic allocation is only enabled for Spark-on-Yarn. Those settings likely have no impact in standalone mode. Nick On Mon, Jan 11, 2016, 5:10 AM Yiannis Gkoufas wrote: > Hi, > > I am exploring a bit the dynamic resource allocation provided by the

Re: efficient checking the existence of an item in a rdd

The key to efficient lookups is having a partitioner in place. If you don't have a partitioner in place, essentially the best you can do is: def contains[T](rdd: RDD[T], value: T): Boolean = ! (rdd.filter(x => x == value).isEmpty) If you are going to do this sort of operation frequently, it

Re: Broadcast var is null

This might seem silly, but... Stop having your object extend App, and instead give it a main method. That's worked for me recently when I've had this issue. (There was a very old issue in Spark related to this; it would seem like a possible regression, if this fixes it for you.) -- Nick On Tue,

Re: GraphX create graph with multiple node attributes

Have you checked to make sure that your hashing function doesn't have any collisions? Node ids have to be unique; so, if you're getting repeated ids out of your hasher, it could certainly lead to dropping of duplicate ids, and therefore loss of vertices. On Sat, Sep 26, 2015 at 10:37 AM JJ

Re: Spark on Yarn: Kryo throws ClassNotFoundException for class included in fat jar

Yes, the jar contains the class: $ jar -tf lumiata-evaluation-assembly-1.0.jar | grep 2028/Document/Document com/i2028/Document/Document$1.class com/i2028/Document/Document.class What else can I do? Is there any way to get more information about the classes available to the particular

Re: Spark on Yarn: Kryo throws ClassNotFoundException for class included in fat jar

> remember stackoverflow has some examples on how to print all classes, but > how to print all classes of kryo classloader - no idea. > > On 8 September 2015 at 16:43, Nick Peterson <nrpeter...@gmail.com> wrote: > >> Yes, the jar contains the class: >> >> $ jar

Re: Spark on Yarn: Kryo throws ClassNotFoundException for class included in fat jar

> > On 8 September 2015 at 18:30, Nick Peterson <nrpeter...@gmail.com> wrote: > >> Yeah... none of the jars listed on the classpath contain this class. The >> only jar that does is the fat jar that I'm submitting with spark-submit, >> which as mentioned isn't