= file.filter( line = line.contains([INFO]) )
You can view the lines using the println method like this:
a1.foreach(println)
You can also count the number of such lines using the count function like
this:
val b1 = file.filter( line = line.contains([ERROR]) ).count()
Regards,
*Ritesh Kumar
Can you please mention the output for the following :
java -version
javac -version
Hi,
I'm using IntelliJ ide for my spark project.
I've compiled spark 1.3.0 for scala 2.11.4 and here's the one of the
compiled jar installed in my m2 folder :
~/.m2/repository/org/apache/spark/spark-core_2.11/1.3.0/spark-core_2.11-1.3.0.jar
But when I add this dependency in my pom file for the
Yes, finally solved. It was there in front of my eyes all time.
Thanks a lot Pete.
Hi,
I'm getting this error while running spark as a java project using maven :
15/06/15 17:11:38 INFO SparkContext: Running Spark version 1.3.0
15/06/15 17:11:38 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/06/15
Hi,
Though my project has nothing to do with akka, I'm getting this error :
Exception in thread main com.typesafe.config.ConfigException$Missing: No
configuration setting found for key 'akka.version'
at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:124)
at
It did hang for me too. High RAM consumption during build. Had to free a
lot of RAM and introduce swap memory just to get it build in my 3rd attempt.
Everything else looks fine. You can download the prebuilt versions from the
Spark homepage to save yourself from all this trouble.
Thanks,
Ritesh
Use this:
sequenceiq/docker
Here's a link to their github repo:
docker-spark https://github.com/sequenceiq/docker-spark
They have repos for other big data tools too which are agin really nice.
Its being maintained properly by their devs and
these should
be harmonized. I imagine one could be excluded; I imagine we could
harmonize the version manually. In practice, I also imagine it doesn't
cause any problem but feel free to propose a fix along those lines.
On Thu, Apr 9, 2015 at 4:54 PM, Ritesh Kumar Singh
riteshoneinamill
. They
aren't because of unnecessary dependencies.
On Thu, Apr 9, 2015 at 5:14 PM, Ritesh Kumar Singh
riteshoneinamill...@gmail.com wrote:
Though the warnings can be ignored, they add up in the log files while
compiling other projects too. And there are a lot of those warnings. Any
workaround? How
Hi,
Are there any tutorials that explains all the changelogs between Spark
0.8.0 and Spark 1.3.0 and how can we approach this issue.
try using breeze (scala linear algebra library)
On Fri, Feb 27, 2015 at 5:56 PM, shahab shahab.mok...@gmail.com wrote:
Thanks a lot Vijay, let me see how it performs.
Best
Shahab
On Friday, February 27, 2015, Vijay Saraswat vi...@saraswat.org wrote:
Available in GML --
How did you build your spark 1.1.1 ?
On Wed, Dec 10, 2014 at 10:41 AM, amin mohebbi aminn_...@yahoo.com.invalid
wrote:
I'm trying to build a very simple scala standalone app using the Mllib,
but I get the following error when trying to bulid the program:
Object mllib is not a member of
On a rough note,
Step 1: Install Hadoop2.x in all the machines on cluster
Step 2: Check if Hadoop cluster is working
Step 3: Setup Apache Spark as given on the documentation page for the
cluster.
Check the status of cluster on the master UI
As it is some data mining project, configure Hive too.
For converting an Array or any List to a RDD, we can try using :
sc.parallelize(groupedScore)//or whatever the name of the list
variable is
On Mon, Dec 1, 2014 at 8:14 PM, Xuefeng Wu ben...@gmail.com wrote:
Hi, I have a problem, it is easy in Scala code, but I can not take the top
N
Spark configuration settings can be found here
http://spark.apache.org/docs/latest/configuration.html
Hope it helps :)
On Sun, Nov 30, 2014 at 9:55 PM, Brian Dolan buddha_...@yahoo.com.invalid
wrote:
Howdy Folks,
What is the correct syntax in 1.0.0 to set networking variables in spark
As Marcelo mentioned, the issue occurs mostly when incompatible classes are
used by executors or drivers. Try out if the output is coming on
spark-shell. If yes, then most probably in your case, there might be some
issue with your configuration files. It will be helpful if you can paste
the
Not sure how to solve this, but spotted these lines in the logs:
14/11/18 14:28:23 INFO YarnAllocationHandler: Container marked as
*failed*: container_1415961020140_0325_01_02
14/11/18 14:28:38 INFO YarnAllocationHandler: Container marked as
*failed*: container_1415961020140_0325_01_03
It can be a serialization issue.
Happens when there are different versions installed on the same system.
What do you mean by the first time you installed and tested it out?
On Wed, Nov 19, 2014 at 3:29 AM, Anson Abraham anson.abra...@gmail.com
wrote:
I'm essentially loading a file and saving
My sbt file for the project includes this:
libraryDependencies ++= Seq(
org.apache.spark %% spark-core % 1.1.0,
org.apache.spark %% spark-mllib % 1.1.0,
org.apache.commons % commons-math3 % 3.3
)
=
Still I am
with Double).
On 17 November 2014 00:14, Ritesh Kumar Singh
riteshoneinamill...@gmail.com wrote:
Hi,
I have a method that returns DenseMatrix:
def func(str: String): DenseMatrix = {
...
...
}
But I keep getting this error:
*class DenseMatrix takes type parameters*
I tried
Hi,
I have a method that returns DenseMatrix:
def func(str: String): DenseMatrix = {
...
...
}
But I keep getting this error:
*class DenseMatrix takes type parameters*
I tried this too:
def func(str: String): DenseMatrix(Int, Int, Array[Double]) = {
...
...
}
But this gives me
Yes... found the output on web UI of the slave.
Thanks :)
On Tue, Nov 11, 2014 at 2:48 AM, Ankur Dave ankurd...@gmail.com wrote:
At 2014-11-10 22:53:49 +0530, Ritesh Kumar Singh
riteshoneinamill...@gmail.com wrote:
Tasks are now getting submitted, but many tasks don't happen.
Like, after
go to your spark home and then into the conf/ directory and then edit the
log4j.properties file i.e. :
gedit $SPARK_HOME/conf/log4j.properties
and set root logger to:
log4j.rootCategory=WARN, console
U don't need to build spark for the changes to take place. Whenever you
open spark-shel, it
-- Forwarded message --
From: Ritesh Kumar Singh riteshoneinamill...@gmail.com
Date: Tue, Nov 11, 2014 at 2:18 PM
Subject: Re: disable log4j for spark-shell
To: lordjoe lordjoe2...@gmail.com
Cc: u...@spark.incubator.apache.org
go to your spark home and then into the conf
We have RDD.saveAsTextFile and RDD.saveAsObjectFile for saving the output
to any location specified. The params to be provided are:
path of storage location
no. of partitions
For giving an hdfs path we use the following format:
/user/user-name/directory-to-sore/
On Tue, Nov 11, 2014 at 6:28 PM,
There is a property :
spark.ui.killEnabled
which needs to be set true for killing applications directly from the webUI.
Check the link:
Kill Enable spark job
http://spark.apache.org/docs/latest/configuration.html#spark-ui
Thanks
On Tue, Nov 11, 2014 at 7:42 PM, Sonal Goyal
Never tried this form but just guessing,
What's the output when you submit this jar: \\shares\publish\Spark\app1\
someJar.jar
using spark-submit.cmd
How can I remove all the INFO logs that appear on the console when I submit
an application using spark-submit?
Kumar Singh
riteshoneinamill...@gmail.com wrote:
How can I remove all the INFO logs that appear on the console when I
submit an application using spark-submit?
On Mon, Nov 10, 2014 at 10:52 PM, Ritesh Kumar Singh
riteshoneinamill...@gmail.com wrote:
Tasks are now getting submitted, but many tasks don't happen.
Like, after opening the spark-shell, I load a text file from disk and try
printing its contentsas:
sc.textFile(/path/to/file).foreach
-- Forwarded message --
From: Ritesh Kumar Singh riteshoneinamill...@gmail.com
Date: Mon, Nov 10, 2014 at 10:52 PM
Subject: Re: Executor Lost Failure
To: Akhil Das ak...@sigmoidanalytics.com
Tasks are now getting submitted, but many tasks don't happen.
Like, after opening
32 matches
Mail list logo
