On 25 Mar 2015, at 21:54, roni
roni.epi...@gmail.commailto:roni.epi...@gmail.com wrote:
Is there any way that I can install the new one and remove previous version.
I installed spark 1.3 on my EC2 master and set teh spark home to the new one.
But when I start teh spark-shell I get -
I have a EC2 cluster created using spark version 1.2.1.
And I have a SBT project .
Now I want to upgrade to spark 1.3 and use the new features.
Below are issues .
Sorry for the long post.
Appreciate your help.
Thanks
-Roni
Question - Do I have to create a new cluster using spark 1.3?
Here is
For the Spark SQL parts, 1.3 breaks backwards compatibility, because before
1.3, Spark SQL was considered experimental where API changes were allowed.
So, H2O and ADA compatible with 1.2.X might not work with 1.3.
dean
Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
What version of Spark do the other dependencies rely on (Adam and H2O?) - that
could be it
Or try sbt clean compile
—
Sent from Mailbox
On Wed, Mar 25, 2015 at 5:58 PM, roni roni.epi...@gmail.com wrote:
I have a EC2 cluster created using spark version 1.2.1.
And I have a SBT project .
Even if H2o and ADA are dependent on 1.2.1 , it should be backword
compatible, right?
So using 1.3 should not break them.
And the code is not using the classes from those libs.
I tried sbt clean compile .. same errror
Thanks
_R
On Wed, Mar 25, 2015 at 9:26 AM, Nick Pentreath
Ah I see now you are trying to use a spark 1.2 cluster - you will need to be
running spark 1.3 on your EC2 cluster in order to run programs built against
spark 1.3.
You will need to terminate and restart your cluster with spark 1.3
—
Sent from Mailbox
On Wed, Mar 25, 2015 at 6:39 PM,
Thanks Dean and Nick.
So, I removed the ADAM and H2o from my SBT as I was not using them.
I got the code to compile - only for fail while running with -
SparkContext: Created broadcast 1 from textFile at kmerIntersetion.scala:21
Exception in thread main java.lang.NoClassDefFoundError:
Weird. Are you running using SBT console? It should have the spark-core jar
on the classpath. Similarly, spark-shell or spark-submit should work, but
be sure you're using the same version of Spark when running as when
compiling. Also, you might need to add spark-sql to your SBT dependencies,
but
Yes, that's the problem. The RDD class exists in both binary jar files, but
the signatures probably don't match. The bottom line, as always for tools
like this, is that you can't mix versions.
Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
My cluster is still on spark 1.2 and in SBT I am using 1.3.
So probably it is compiling with 1.3 but running with 1.2 ?
On Wed, Mar 25, 2015 at 12:34 PM, Dean Wampler deanwamp...@gmail.com
wrote:
Weird. Are you running using SBT console? It should have the spark-core
jar on the classpath.
Is there any way that I can install the new one and remove previous version.
I installed spark 1.3 on my EC2 master and set teh spark home to the new
one.
But when I start teh spark-shell I get -
java.lang.UnsatisfiedLinkError:
org.apache.hadoop.security.JniBasedUnixGroupsMapping.anchorNative()V
You could stop the running the processes and run the same processes using
the new version, starting with the master and then the slaves. You would
have to snoop around a bit to get the command-line arguments right, but
it's doable. Use `ps -efw` to find the command-lines used. Be sure to rerun
Marcelo’s work-around works. So if you are using the itemsimilarity stuff, the
CLI has a way to solve the class not found and I can point out how to do the
equivalent if you are using the library API. Ping me if you care.
On Feb 28, 2015, at 2:27 PM, Erlend Hamnaberg erl...@hamnaberg.net
Maybe but any time the work around is to use spark-submit --conf
spark.executor.extraClassPath=/guava.jar blah” that means that standalone apps
must have hard coded paths that are honored on every worker. And as you know a
lib is pretty much blocked from use of this version of Spark—hence the
Yes. I ran into this problem with mahout snapshot and spark 1.2.0 not
really trying to figure out why that was a problem, since there were
already too many moving parts in my app. Obviously there is a classpath
issue somewhere.
/Erlend
On 27 Feb 2015 22:30, Pat Ferrel p...@occamsmachete.com
This seems like a job for userClassPathFirst. Or could be. It's
definitely an issue of visibility between where the serializer is and
where the user class is.
At the top you said Pat that you didn't try this, but why not?
On Fri, Feb 27, 2015 at 10:11 PM, Pat Ferrel p...@occamsmachete.com wrote:
I understand that I need to supply Guava to Spark. The HashBiMap is created in
the client and broadcast to the workers. So it is needed in both. To achieve
this there is a deps.jar with Guava (and Scopt but that is only for the
client). Scopt is found so I know the jar is fine for the client.
Ah, I see. That makes a lot of sense now.
You might be running into some weird class loader visibility issue.
I've seen some bugs in jira about this in the past, maybe you're
hitting one of them.
Until I have some time to investigate (of if you're curious feel free
to scavenge jira), a
Hi.
I have had a simliar issue. I had to pull the JavaSerializer source into my
own project, just so I got the classloading of this class under control.
This must be a class loader issue with spark.
-E
On Fri, Feb 27, 2015 at 8:52 PM, Pat Ferrel p...@occamsmachete.com wrote:
I understand
On Fri, Feb 27, 2015 at 1:30 PM, Pat Ferrel p...@occamsmachete.com wrote:
@Marcelo do you mean by modifying spark.executor.extraClassPath on all
workers, that didn’t seem to work?
That's an app configuration, not a worker configuration, so if you're
trying to set it on the worker configuration
I don’t use spark-submit I have a standalone app.
So I guess you want me to add that key/value to the conf in my code and make
sure it exists on workers.
On Feb 27, 2015, at 1:47 PM, Marcelo Vanzin van...@cloudera.com wrote:
On Fri, Feb 27, 2015 at 1:42 PM, Pat Ferrel p...@occamsmachete.com
@Erlend hah, we were trying to merge your PR and ran into this—small world. You
actually compile the JavaSerializer source in your project?
@Marcelo do you mean by modifying spark.executor.extraClassPath on all workers,
that didn’t seem to work?
On Feb 27, 2015, at 1:23 PM, Erlend Hamnaberg
On Fri, Feb 27, 2015 at 1:42 PM, Pat Ferrel p...@occamsmachete.com wrote:
I changed in the spark master conf, which is also the only worker. I added a
path to the jar that has guava in it. Still can’t find the class.
Sorry, I'm still confused about what config you're changing. I'm
suggesting
Thanks! that worked.
On Feb 27, 2015, at 1:50 PM, Pat Ferrel p...@occamsmachete.com wrote:
I don’t use spark-submit I have a standalone app.
So I guess you want me to add that key/value to the conf in my code and make
sure it exists on workers.
On Feb 27, 2015, at 1:47 PM, Marcelo Vanzin
I’ll try to find a Jira for it. I hope a fix is in 1.3
On Feb 27, 2015, at 1:59 PM, Pat Ferrel p...@occamsmachete.com wrote:
Thanks! that worked.
On Feb 27, 2015, at 1:50 PM, Pat Ferrel p...@occamsmachete.com wrote:
I don’t use spark-submit I have a standalone app.
So I guess you want me to
I changed in the spark master conf, which is also the only worker. I added a
path to the jar that has guava in it. Still can’t find the class.
Trying Erland’s idea next.
On Feb 27, 2015, at 1:35 PM, Marcelo Vanzin van...@cloudera.com wrote:
On Fri, Feb 27, 2015 at 1:30 PM, Pat Ferrel
Getting an error that confuses me. Running a largish app on a standalone
cluster on my laptop. The app uses a guava HashBiMap as a broadcast value. With
Spark 1.1.0 I simply registered the class and its serializer with kryo like
this:
Could this be caused by Spark using shaded Guava jar ?
Cheers
On Wed, Feb 25, 2015 at 3:26 PM, Pat Ferrel p...@occamsmachete.com wrote:
Getting an error that confuses me. Running a largish app on a standalone
cluster on my laptop. The app uses a guava HashBiMap as a broadcast value.
With
28 matches
Mail list logo
