Just launched an EC2 cluster from git hash
9fe693b5b6ed6af34ee1e800ab89c8a11991ea38. Calling take() on an RDD
accessing data in S3 yields the following error output.
I understand that NoClassDefFoundError errors may mean something in the
deployment was messed up. Is that correct? When I launch a
This one is typically due to a mismatch between the Hadoop versions --
i.e., Spark is compiled against 1.0.4 but is running with 2.3.0 in the
classpath, or something like that. Not certain why you're seeing this with
spark-ec2, but I'm assuming this is related to the issues you posted in a
My guess is that this is related to
https://issues.apache.org/jira/browse/SPARK-2471 where the S3 library gets
excluded from the SBT assembly jar. I am not sure if the assembly jar used
in EC2 is generated using SBT though.
Shivaram
On Mon, Jul 14, 2014 at 10:02 PM, Aaron Davidson
Yeah - this is likely caused by SPARK-2471.
On Mon, Jul 14, 2014 at 10:11 PM, Shivaram Venkataraman
shiva...@eecs.berkeley.edu wrote:
My guess is that this is related to
https://issues.apache.org/jira/browse/SPARK-2471 where the S3 library gets
excluded from the SBT assembly jar. I am not sure
Okie doke--added myself as a watcher on that issue.
On a related note, what are the thoughts on automatically spinning up/down
EC2 clusters and running tests against them? It would probably be way too
cumbersome to do that for every build, but perhaps on some schedule it
could help validate that
