Missed the $
export
SPARK_CLASSPATH=/home/christophe/Development/spark-streaming3/config-1.2.1.jar:
*$SPARK_CLASSPATH*
Thanks
Best Regards
On Fri, Jan 2, 2015 at 4:57 PM, Akhil Das ak...@sigmoidanalytics.com
wrote:
Can you try:
export
Thank you Akhil for your idea.
In spark-env.sh, I set
export
SPARK_CLASSPATH=/home/christophe/Development/spark-streaming3/config-1.2.1.jar
When I run bin/compute-classpath.sh
I get Spark's classpath:
Hi,
I am running spark 1.1.0 on yarn. I have custom set of modules installed under
same location on each executor node and wondering how can I pass the executors
the PYTHONPATH so that they can use the modules.
I've tried this:
spark-env.sh:export PYTHONPATH=/tmp/test/
Hi all,
I'm curious about MLlib and if it is possible to do incremental training on
the ALSModel.
Usually training is run first, and then you can query. But in my case, data
is collected in real-time and I want the predictions of my ALSModel to
consider the latest data without complete
Do you have some example code of what you are trying to do?
Robin
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/DAG-info-tp20940p20941.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Hi All -
Is JdbcRdd currently supported? Having trouble finding any info or
examples?
ok, I see now what's happening - the pkg.mod.test is serialized by reference
and there is nothing actually trying to import pkg.mod on the executors so the
reference is broken.
so how can I get the pkg.mod imported on the executors?
thanks,Antony.
On Friday, 2 January 2015, 13:49, Antony
Hi, guys
I tried to run job of spark streaming with kafka on YARN.
My business logic is very simple.
Just listen on kafka topic and write dstream to hdfs on each batch
iteration.
After launching streaming job few hours, it works well. However suddenly
died by ResourceManager.
ResourceManager
Hi
I have compiled OpenBlas library into nonstandard directory and I want to
inform Spark app about it via:
-Dcom.github.fommil.netlib.NativeSystemBLAS.natives=/usr/local/lib/libopenblas.so
which is a standard option in netlib-java
(https://github.com/fommil/netlib-java)
I tried 2 ways:
1. via
Thanks everyone. I studied the source code and realized minPartitions is
passed over to Hadoop's InputFormat and its upto the InputFormat
implementation to use the parameter as a hint.
Thanks,
Aniket
On Fri, Jan 2, 2015, 7:13 AM Rishi Yadav ri...@infoobjects.com wrote:
Hi Ankit,
Optional
Hi, guys
I tried to run job of spark streaming with kafka on YARN.
My business logic is very simple.
Just listen on kafka topic and write dstream to hdfs on each batch iteration.
After launching streaming job few hours, it works well. However suddenly died
by ResourceManager.
ResourceManager
do you assemble the uber jar ?
you can use sbt assembly to build the jar and then run. It should fix the
issue
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/NoClassDefFoundError-when-trying-to-run-spark-application-tp20707p20944.html
Sent from the Apache
i noticed spark 1.2.0 bumps the akka version. since spark uses it's own
akka version, does this mean it can co-exist with another akka version in
the same JVM? has anyone tried this?
we have some spark apps that also use akka (2.2.3) and spray. if different
akka versions causes conflicts then
There is no need to include user@spark.apache.org in subscription request.
FYI
On Fri, Jan 2, 2015 at 7:36 AM, Pankaj pankajnaran...@gmail.com wrote:
Agreed with Jerry. Aside from Tachyon, seeing this for general debugging
would be very helpful.
Haoyuan, is that feature you are referring to related to
https://issues.apache.org/jira/browse/SPARK-975?
In the interim, I've found the toDebugString() method useful (but it
renders execution as a
Looking a little closer @ the launch_container.sh file, it appears to be
adding a $PWD/__app__.jar to the classpath but there is no __app__.jar in
the directory pointed to by PWD. Any ideas?
On Fri, Jan 2, 2015 at 4:20 PM, Corey Nolet cjno...@gmail.com wrote:
I'm trying to get a SparkContext
I'm trying to get a SparkContext going in a web container which is being
submitted through yarn-client. I'm trying two different approaches and both
seem to be resulting in the same error from the yarn nodemanagers:
1) I'm newing up a spark context direct, manually adding all the lib jars
from
Jerry,
Great question. Spark and Tachyon capture lineage information at different
granularities. We are working on an integration between Spark/Tachyon about
this. Hope to get it ready to be released soon.
Best,
Haoyuan
On Fri, Jan 2, 2015 at 12:24 PM, Jerry Lam chiling...@gmail.com wrote:
Hi,
New to spark so just feeling my way in using it on a standalone server under
linux.
I'm using scala to store running count totals of certain tokens in my
streaming data and publishing a top 10 list.
eg
(TokenX,count)
(TokenY,count)
..
At the moment this is just being printed to std out
Hi spark developers,
I was thinking it would be nice to extract the data lineage information
from a data processing pipeline. I assume that spark/tachyon keeps this
information somewhere. For instance, a data processing pipeline uses
datasource A and B to produce C. C is then used by another
yeah.. i went through the source, and unless i'm missing something it's
not.. agreed, i'd love to see it implemented!
On Fri, Jan 2, 2015 at 3:59 PM, Tim Schweichler
tim.schweich...@healthination.com wrote:
Doesn't look like it is at the moment. If that's the case I'd love to
see it
Well that's confusing. I have the same issue. So you're saying I have to
compile Spark with Yarn set to true to make it work with Hadoop 2.2.0 in
Standalone mode?
--
View this message in context:
Doesn't look like it is at the moment. If that's the case I'd love to see it
implemented.
From: elliott cordo elliottco...@gmail.commailto:elliottco...@gmail.com
Date: Friday, January 2, 2015 at 8:17 AM
To: user@spark.apache.orgmailto:user@spark.apache.org
So looking @ the actual code- I see where it looks like --class 'notused'
--jar null is set on the ClientBase.scala when yarn is being run in client
mode. One thing I noticed is that the jar is being set by trying to grab
the jar's uri from the classpath resources- in this case I think it's
Like before I get a java.lang.NoClassDefFoundError:
akka/stream/FlowMaterializer$
This can be solved using assembly plugin. you need to enable assembly plugin
in global plugins
C:\Users\infoshore\.sbt\0.13\plugins
add a line in plugins.sbt addSbtPlugin(com.eed3si9n % sbt-assembly %
0.11.0)
.. and looking even further, it looks like the actual command tha'ts
executed starting up the JVM to run the
org.apache.spark.deploy.yarn.ExecutorLauncher is passing in --class
'notused' --jar null.
I would assume this isn't expected but I don't see where to set these
properties or why they
There is a JIRA for it: https://issues.apache.org/jira/browse/SPARK-4981
On Fri, Jan 2, 2015 at 8:28 PM, Peng Cheng rhw...@gmail.com wrote:
I was under the impression that ALS wasn't designed for it :- The famous
ebay online recommender uses SGD
However, you can try using the previous model
I was under the impression that ALS wasn't designed for it :- The famous
ebay online recommender uses SGD
However, you can try using the previous model as starting point, and
gradually reduce the number of iteration after the model stablize. I never
verify this idea, so you need to at least
Try and see if this helps. http://zeppelin-project.org/
-Sathish
On Fri Jan 02 2015 at 8:20:54 PM Pankaj Narang pankajnaran...@gmail.com
wrote:
Thomus,
Spark does not provide any web interface directly. There might be third
party apps providing dashboards
but I am not aware of any for the
Thomus,
Spark does not provide any web interface directly. There might be third
party apps providing dashboards
but I am not aware of any for the same purpose.
*You can use some methods so that this data is saved on file system instead
of being printed on screen
Some of the methods you can use
Please see http://search-hadoop.com/m/JW1q53L9PJ
On Fri, Jan 2, 2015 at 4:31 PM, RP hadoo...@outlook.com wrote:
Hello Guys,
Spark noob here. I am trying to create RDD from JSON data fetched from
URL parsing.
My URL parsing function gives me JSON in string format.
How do I convert JSON
Hello Guys,
Spark noob here. I am trying to create RDD from JSON data fetched
from URL parsing.
My URL parsing function gives me JSON in string format.
How do I convert JSON string to JSONRDD so that I can use it in SparkSQL.
// get json data in string fromat
val jsonURLData =
OK this is how I solved it. Not elegant at all but works and I need to move
ahead at this time.Converting to pair RDD is now not required.
reacRdd.map(line = line.split(',')).map(fields = {
if (fields.length = 10 !fields(0).contains(VAERS_ID)) {
Most of the time a NoSuchMethodError means wrong classpath settings, and
some jar file is overriden by a wrong version. In your case it could be
netty.
On 1/3/15 1:36 PM, Niranda Perera wrote:
Hi all,
I am evaluating the spark sources API released with Spark 1.2.0. But
I'm getting a
Hi all,
I am evaluating the spark sources API released with Spark 1.2.0. But I'm
getting a ava.lang.NoSuchMethodError:
org.jboss.netty.channel.socket.nio.NioWorkerPool.init(Ljava/util/concurrent/Executor;I)V
error running the program.
Error log:
15/01/03 10:41:30 ERROR ActorSystemImpl: Uncaught
Please see http://akka.io/news/2014/05/22/akka-2.3.3-released.html which
points to
http://doc.akka.io/docs/akka/2.3.3/project/migration-guide-2.2.x-2.3.x.html?_ga=1.35212129.1385865413.1420220234
Cheers
On Fri, Jan 2, 2015 at 9:11 AM, Koert Kuipers ko...@tresata.com wrote:
i noticed spark
Hi,
I get the following error when I build spark using sbt:
[error] Nonzero exit code (128): git clone
https://github.com/ScrapCodes/sbt-pom-reader.git
/home/karthik/.sbt/0.13/staging/ad8e8574a5bcb2d22d23/sbt-pom-reader
[error] Use 'last' for the full log.
Any help please?
38 matches
Mail list logo
