Hey all,
We're working on setting up a Spark 1.6.1 cluster on Amazon EC2, and
encountering some problems related to pre-emption. We have followed all
the instructions for setting up dynamic allocation, including enabling the
external spark shuffle service in the YARN NodeManagers.
When a contain
My understanding is that dynamic allocation is only enabled for
Spark-on-Yarn. Those settings likely have no impact in standalone mode.
Nick
On Mon, Jan 11, 2016, 5:10 AM Yiannis Gkoufas wrote:
> Hi,
>
> I am exploring a bit the dynamic resource allocation provided by the
> Standalone Cluster M
The key to efficient lookups is having a partitioner in place.
If you don't have a partitioner in place, essentially the best you can do
is:
def contains[T](rdd: RDD[T], value: T): Boolean = ! (rdd.filter(x => x ==
value).isEmpty)
If you are going to do this sort of operation frequently, it might
This might seem silly, but...
Stop having your object extend App, and instead give it a main method.
That's worked for me recently when I've had this issue. (There was a very
old issue in Spark related to this; it would seem like a possible
regression, if this fixes it for you.)
-- Nick
On Tue,
Have you checked to make sure that your hashing function doesn't have any
collisions? Node ids have to be unique; so, if you're getting repeated ids
out of your hasher, it could certainly lead to dropping of duplicate ids,
and therefore loss of vertices.
On Sat, Sep 26, 2015 at 10:37 AM JJ wrote
t can contain
>>> this class by any chance
>>>
>>> regarding your question about classloader - no idea, probably there is,
>>> I remember stackoverflow has some examples on how to print all classes, but
>>> how to print all classes of kryo classloader - no i
ckoverflow has some examples on how to print all classes, but
> how to print all classes of kryo classloader - no idea.
>
> On 8 September 2015 at 16:43, Nick Peterson wrote:
>
>> Yes, the jar contains the class:
>>
>> $ jar -tf lumiata-evaluation-assembly-1.0.jar | gr
Yes, the jar contains the class:
$ jar -tf lumiata-evaluation-assembly-1.0.jar | grep 2028/Document/Document
com/i2028/Document/Document$1.class
com/i2028/Document/Document.class
What else can I do? Is there any way to get more information about the
classes available to the particular classloade