Re: Is there any Sparse Matrix implementation in Spark/MLib?

try using breeze (scala linear algebra library) On Fri, Feb 27, 2015 at 5:56 PM, shahab wrote: > Thanks a lot Vijay, let me see how it performs. > > Best > Shahab > > > On Friday, February 27, 2015, Vijay Saraswat wrote: > >> Available in GML -- >> >> http://x10-lang.org/x10-community/applicati

Re: Official Docker container for Spark

Use this: sequenceiq/docker Here's a link to their github repo: docker-spark They have repos for other big data tools too which are agin really nice. Its being maintained properly by their devs and

Re: Recommended Scala version

Yes, recommended version is 2.10 as all the features are not supported by 2.11 version. Kafka libraries and JDBC components are yet to be ported to 2.11 version. And so if your project doesn't depend on these components, you can give v2.11 a try. Here's a link

Re: Can't build Spark 1.3

It did hang for me too. High RAM consumption during build. Had to free a lot of RAM and introduce swap memory just to get it build in my 3rd attempt. Everything else looks fine. You can download the prebuilt versions from the Spark homepage to save yourself from all this trouble. Thanks, Ritesh

Error using spark 1.3.0 with maven

Hi, I'm getting this error while running spark as a java project using maven : 15/06/15 17:11:38 INFO SparkContext: Running Spark version 1.3.0 15/06/15 17:11:38 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/06/15 1

akka configuration not found

Hi, Though my project has nothing to do with akka, I'm getting this error : Exception in thread "main" com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'akka.version' at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:124) at com.typesafe.config.

Local spark jars not being detected

Hi, I'm using IntelliJ ide for my spark project. I've compiled spark 1.3.0 for scala 2.11.4 and here's the one of the compiled jar installed in my m2 folder : ~/.m2/repository/org/apache/spark/spark-core_2.11/1.3.0/spark-core_2.11-1.3.0.jar But when I add this dependency in my pom file for the p

Re: Local spark jars not being detected

Yes, finally solved. It was there in front of my eyes all time. Thanks a lot Pete.

Migrating from Spark 0.8.0 to Spark 1.3.0

Hi, Are there any tutorials that explains all the changelogs between Spark 0.8.0 and Spark 1.3.0 and how can we approach this issue.

Overlapping classes warnings

Hi, During compilation I get a lot of these: [WARNING] kryo-2.21.jar, reflectasm-1.07-shaded.jar define 23 overlappping classes: [WARNING] commons-beanutils-1.7.0.jar, commons-beanutils-core-1.8.0.jar define 82 overlappping classes: [WARNING] commons-beanutils-1.7.0.jar, commons-collections-3

Re: Overlapping classes warnings

hough these should > be harmonized. I imagine one could be excluded; I imagine we could > harmonize the version manually. In practice, I also imagine it doesn't > cause any problem but feel free to propose a fix along those lines. > > On Thu, Apr 9, 2015 at 4:54 PM, Ritesh Kumar

Re: Overlapping classes warnings

en't because of unnecessary dependencies. > > On Thu, Apr 9, 2015 at 5:14 PM, Ritesh Kumar Singh > wrote: > > Though the warnings can be ignored, they add up in the log files while > > compiling other projects too. And there are a lot of those warnings. Any > > work

Re: Unsupported major.minor version 51.0

Can you please mention the output for the following : java -version javac -version

Re: Feasibility Project - Text Processing and Category Classification

ion like this: > > val b1 = file.filter( line => line.contains("[ERROR]") ).count() Regards, > *Ritesh Kumar Singh,**https://riteshtoday.wordpress.com/ > <https://riteshtoday.wordpress.com/>*

Removing INFO logs

How can I remove all the INFO logs that appear on the console when I submit an application using spark-submit?

Re: Removing INFO logs

> On Mon, Nov 10, 2014 at 8:21 PM, Ritesh Kumar Singh < > riteshoneinamill...@gmail.com> wrote: > >> How can I remove all the INFO logs that appear on the console when I >> submit an application using spark-submit? >> > >

Executor Lost Failure

Hi, I am trying to submit my application using spark-submit, using following spark-default.conf params: spark.master spark://:7077 spark.eventLog.enabled true spark.serializer org.apache.spark.serializer.KryoSerializer spark.executor.extraJavaOptions

Re: Executor Lost Failure

On Mon, Nov 10, 2014 at 10:52 PM, Ritesh Kumar Singh < riteshoneinamill...@gmail.com> wrote: > Tasks are now getting submitted, but many tasks don't happen. > Like, after opening the spark-shell, I load a text file from disk and try > printing its contentsas: > > &

Fwd: Executor Lost Failure

-- Forwarded message -- From: Ritesh Kumar Singh Date: Mon, Nov 10, 2014 at 10:52 PM Subject: Re: Executor Lost Failure To: Akhil Das Tasks are now getting submitted, but many tasks don't happen. Like, after opening the spark-shell, I load a text file from disk and try pri

Re: Fwd: Executor Lost Failure

Yes... found the output on web UI of the slave. Thanks :) On Tue, Nov 11, 2014 at 2:48 AM, Ankur Dave wrote: > At 2014-11-10 22:53:49 +0530, Ritesh Kumar Singh < > riteshoneinamill...@gmail.com> wrote: > > Tasks are now getting submitted, but many tasks don't happen.

Re: disable log4j for spark-shell

go to your spark home and then into the conf/ directory and then edit the log4j.properties file i.e. : >gedit $SPARK_HOME/conf/log4j.properties and set root logger to: log4j.rootCategory=WARN, console U don't need to build spark for the changes to take place. Whenever you open spark-shel, it

Fwd: disable log4j for spark-shell

-- Forwarded message -- From: Ritesh Kumar Singh Date: Tue, Nov 11, 2014 at 2:18 PM Subject: Re: disable log4j for spark-shell To: lordjoe Cc: u...@spark.incubator.apache.org go to your spark home and then into the conf/ directory and then edit the log4j.properties file i.e

Re: save as file

We have RDD.saveAsTextFile and RDD.saveAsObjectFile for saving the output to any location specified. The params to be provided are: >path of storage location >no. of partitions For giving an hdfs path we use the following format: "/user///" On Tue, Nov 11, 2014 at 6:28 PM, Naveen Kumar Pokala < n

Re: How to kill a Spark job running in cluster mode ?

There is a property : spark.ui.killEnabled which needs to be set true for killing applications directly from the webUI. Check the link: Kill Enable spark job Thanks On Tue, Nov 11, 2014 at 7:42 PM, Sonal Goyal wrote: > The web

Re: Spark-submit and Windows / Linux mixed network

Never tried this form but just guessing, What's the output when you submit this jar: \\shares\publish\Spark\app1\ someJar.jar using spark-submit.cmd

Re: How to kill a Spark job running in cluster mode ?

ark.deploy.Client kill " if you > know the driver id. > > 2014-11-11 22:35 GMT+08:00 Ritesh Kumar Singh < > riteshoneinamill...@gmail.com>: > >> There is a property : >>spark.ui.killEnabled >> which needs to be set true for killing applications direct

Returning breeze.linalg.DenseMatrix from method

Hi, I have a method that returns DenseMatrix: def func(str: String): DenseMatrix = { ... ... } But I keep getting this error: *class DenseMatrix takes type parameters* I tried this too: def func(str: String): DenseMatrix(Int, Int, Array[Double]) = { ... ... } But this gives me th

RandomGenerator class not found exception

My sbt file for the project includes this: libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % "1.1.0", "org.apache.spark" %% "spark-mllib" % "1.1.0", "org.apache.commons" % "commons-math3" % "3.3" ) =

Re: RandomGenerator class not found exception

3/3.3 in class path while submitting jar to spark > cluster. Like.. > spark-submit --driver-class-path maths3.3jar --class MainClass --master > spark cluster url appjar > > On Mon, Nov 17, 2014 at 1:55 PM, Ritesh Kumar Singh [via Apache Spark User > List] <[hidden email] <http://use

Re: Returning breeze.linalg.DenseMatrix from method

'm also assuming that SomeOtherType has no subtype > relations with Double). > > On 17 November 2014 00:14, Ritesh Kumar Singh < > riteshoneinamill...@gmail.com> wrote: > >> Hi, >> >> I have a method that returns DenseMatrix: >> def func(str

Re: Spark On Yarn Issue: Initial job has not accepted any resources

Not sure how to solve this, but spotted these lines in the logs: 14/11/18 14:28:23 INFO YarnAllocationHandler: Container marked as *failed*: container_1415961020140_0325_01_02 14/11/18 14:28:38 INFO YarnAllocationHandler: Container marked as *failed*: container_1415961020140_0325_01_03 A

Re: spark-shell giving me error of unread block data

It can be a serialization issue. Happens when there are different versions installed on the same system. What do you mean by the first time you installed and tested it out? On Wed, Nov 19, 2014 at 3:29 AM, Anson Abraham wrote: > I'm essentially loading a file and saving output to another locatio

Re: spark-shell giving me error of unread block data

As Marcelo mentioned, the issue occurs mostly when incompatible classes are used by executors or drivers. Try out if the output is coming on spark-shell. If yes, then most probably in your case, there might be some issue with your configuration files. It will be helpful if you can paste the conten

Re: Setting network variables in spark-shell

Spark configuration settings can be found here Hope it helps :) On Sun, Nov 30, 2014 at 9:55 PM, Brian Dolan wrote: > Howdy Folks, > > What is the correct syntax in 1.0.0 to set networking variables in spark > shell? Specifically, I'd li

Re: How take top N of top M from RDD as RDD

For converting an Array or any List to a RDD, we can try using : >sc.parallelize(groupedScore)//or whatever the name of the list variable is On Mon, Dec 1, 2014 at 8:14 PM, Xuefeng Wu wrote: > Hi, I have a problem, it is easy in Scala code, but I can not take the top > N from RDD as RDD

Re: Install Apache Spark on a Cluster

On a rough note, Step 1: Install Hadoop2.x in all the machines on cluster Step 2: Check if Hadoop cluster is working Step 3: Setup Apache Spark as given on the documentation page for the cluster. Check the status of cluster on the master UI As it is some data mining project, configure Hive too. Y

Re: Mllib error

How did you build your spark 1.1.1 ? On Wed, Dec 10, 2014 at 10:41 AM, amin mohebbi wrote: > I'm trying to build a very simple scala standalone app using the Mllib, > but I get the following error when trying to bulid the program: > > Object mllib is not a member of package org.apache.spark > >