Marcelo’s work-around works. So if you are using the itemsimilarity stuff, the
CLI has a way to solve the class not found and I can point out how to do the
equivalent if you are using the library API. Ping me if you care.
On Feb 28, 2015, at 2:27 PM, Erlend Hamnaberg erl...@hamnaberg.net
Maybe but any time the work around is to use spark-submit --conf
spark.executor.extraClassPath=/guava.jar blah” that means that standalone apps
must have hard coded paths that are honored on every worker. And as you know a
lib is pretty much blocked from use of this version of Spark—hence the
Yes. I ran into this problem with mahout snapshot and spark 1.2.0 not
really trying to figure out why that was a problem, since there were
already too many moving parts in my app. Obviously there is a classpath
issue somewhere.
/Erlend
On 27 Feb 2015 22:30, Pat Ferrel p...@occamsmachete.com
This seems like a job for userClassPathFirst. Or could be. It's
definitely an issue of visibility between where the serializer is and
where the user class is.
At the top you said Pat that you didn't try this, but why not?
On Fri, Feb 27, 2015 at 10:11 PM, Pat Ferrel p...@occamsmachete.com wrote:
I understand that I need to supply Guava to Spark. The HashBiMap is created in
the client and broadcast to the workers. So it is needed in both. To achieve
this there is a deps.jar with Guava (and Scopt but that is only for the
client). Scopt is found so I know the jar is fine for the client.
Ah, I see. That makes a lot of sense now.
You might be running into some weird class loader visibility issue.
I've seen some bugs in jira about this in the past, maybe you're
hitting one of them.
Until I have some time to investigate (of if you're curious feel free
to scavenge jira), a
Hi.
I have had a simliar issue. I had to pull the JavaSerializer source into my
own project, just so I got the classloading of this class under control.
This must be a class loader issue with spark.
-E
On Fri, Feb 27, 2015 at 8:52 PM, Pat Ferrel p...@occamsmachete.com wrote:
I understand
On Fri, Feb 27, 2015 at 1:30 PM, Pat Ferrel p...@occamsmachete.com wrote:
@Marcelo do you mean by modifying spark.executor.extraClassPath on all
workers, that didn’t seem to work?
That's an app configuration, not a worker configuration, so if you're
trying to set it on the worker configuration
I don’t use spark-submit I have a standalone app.
So I guess you want me to add that key/value to the conf in my code and make
sure it exists on workers.
On Feb 27, 2015, at 1:47 PM, Marcelo Vanzin van...@cloudera.com wrote:
On Fri, Feb 27, 2015 at 1:42 PM, Pat Ferrel p...@occamsmachete.com
@Erlend hah, we were trying to merge your PR and ran into this—small world. You
actually compile the JavaSerializer source in your project?
@Marcelo do you mean by modifying spark.executor.extraClassPath on all workers,
that didn’t seem to work?
On Feb 27, 2015, at 1:23 PM, Erlend Hamnaberg
On Fri, Feb 27, 2015 at 1:42 PM, Pat Ferrel p...@occamsmachete.com wrote:
I changed in the spark master conf, which is also the only worker. I added a
path to the jar that has guava in it. Still can’t find the class.
Sorry, I'm still confused about what config you're changing. I'm
suggesting
Thanks! that worked.
On Feb 27, 2015, at 1:50 PM, Pat Ferrel p...@occamsmachete.com wrote:
I don’t use spark-submit I have a standalone app.
So I guess you want me to add that key/value to the conf in my code and make
sure it exists on workers.
On Feb 27, 2015, at 1:47 PM, Marcelo Vanzin
I’ll try to find a Jira for it. I hope a fix is in 1.3
On Feb 27, 2015, at 1:59 PM, Pat Ferrel p...@occamsmachete.com wrote:
Thanks! that worked.
On Feb 27, 2015, at 1:50 PM, Pat Ferrel p...@occamsmachete.com wrote:
I don’t use spark-submit I have a standalone app.
So I guess you want me to
I changed in the spark master conf, which is also the only worker. I added a
path to the jar that has guava in it. Still can’t find the class.
Trying Erland’s idea next.
On Feb 27, 2015, at 1:35 PM, Marcelo Vanzin van...@cloudera.com wrote:
On Fri, Feb 27, 2015 at 1:30 PM, Pat Ferrel
14 matches
Mail list logo
