Re: Where does the Driver run?

Hello, Is spark.driver.memory per Job or shared across jobs? You should do load testing before setting this? Thanks & regards Arko On Sun, Mar 24, 2019 at 3:09 PM Pat Ferrel wrote: > > 2 Slaves, one of which is also Master. > > Node 1 & 2 are slaves. Node 1 is where I run start-all.sh. > >

Re: Encoder for JValue

ack as JValue (using something like > parse()) > > Thanks, > Muthu > > On Wed, Sep 19, 2018 at 6:35 AM Arko Provo Mukherjee < > arkoprovomukher...@gmail.com> wrote: > >> Hello Spark Gurus, >> >> I am running into an issue with Encoding and wanted

Encoder for JValue

Hello Spark Gurus, I am running into an issue with Encoding and wanted your help. I have a case class with a JObject in it. Ex: *case class SomeClass(a: String, b: JObject)* I also have an encoder for this case class: *val encoder = Encoders.product[**SomeClass**]* Now I am creating a

Re: SparkMaster IP

t HOSTNAME" command-line parameter (the ip > address will be resolved from the supplied HOSTNAME) > > best, > --Jakob > > On Mon, Feb 22, 2016 at 5:09 PM, Arko Provo Mukherjee > <arkoprovomukher...@gmail.com> wrote: >> Hello, >> >> I am running Spark on W

SparkMaster IP

Hello, I am running Spark on Windows. I start up master as follows: .\spark-class.cmd org.apache.spark.deploy.master.Master I see that the SparkMaster doesn't start on 127.0.0.1 but starts on my "actual" IP. This is troublesome for me as I use it in my code and need to change every time I

Re: Submitting Jobs Programmatically

ed Yu <yuzhih...@gmail.com> wrote: > Cycling old bits: > > http://search-hadoop.com/m/q3RTtHrxMj2abwOk2 > > On Fri, Feb 19, 2016 at 6:40 PM, Arko Provo Mukherjee > <arkoprovomukher...@gmail.com> wrote: >> >> Hi, >> >> Thanks for your response. Is

Re: Submitting Jobs Programmatically

& regards Arko On Fri, Feb 19, 2016 at 6:35 PM, Ted Yu <yuzhih...@gmail.com> wrote: > Please see https://spark.apache.org/docs/latest/spark-standalone.html > > On Fri, Feb 19, 2016 at 6:27 PM, Arko Provo Mukherjee > <arkoprovomukher...@gmail.com> wrote: >> >

Re: Submitting Jobs Programmatically

the program? Thanks & regards Arko On Fri, Feb 19, 2016 at 6:01 PM, Holden Karau <hol...@pigscanfly.ca> wrote: > How are you trying to launch your application? Do you have the Spark jars on > your class path? > > > On Friday, February 19, 2016, Arko Provo Mukherjee > <arko

Submitting Jobs Programmatically

Hello, I am trying to submit a spark job via a program. When I run it, I receive the following error: Exception in thread "Thread-1" java.lang.NoClassDefFoundError: org/apache/spark/launcher/SparkLauncher at Spark.SparkConnector.run(MySpark.scala:33) at

Using sbt assembly

Hello, I am trying to use sbt assembly to generate a fat JAR. Here is my \project\assembly.sbt file: resolvers += Resolver.url("bintray-sbt-plugins", url("http://dl.bintray.com/sbt/sbt-plugin-releases;))(Resolver.ivyStylePatterns) addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.13.9")

Re: Spark with .NET

wp4nR1lugin1=+NET+on+Apache+Spark+ > > On Tue, Feb 9, 2016 at 11:43 AM, Arko Provo Mukherjee < > arkoprovomukher...@gmail.com> wrote: > >> Hello, >> >> I want to use Spark (preferable Spark SQL) using C#. Anyone has any >> pointers to that? >> >> Thanks & regards >> Arko >> >> >

Spark with .NET

Hello, I want to use Spark (preferable Spark SQL) using C#. Anyone has any pointers to that? Thanks & regards Arko

Re: Spark with .NET

t templates. >> >> >> >> The code though is available from Mono here: >> https://github.com/mono/mono/tree/master/mcs/class/System.Data.DataSetExtensions >> >> >> >> *From: *Ted Yu <yuzhih...@gmail.com> >> *Date: *Tuesday, Febru

Using GraphX with Spark Streaming?

Hello Spark Gurus, I am trying to learn Spark. I am specially interested in GraphX. Since Spark can used in streaming context as well, I wanted to know whether it is possible to use the Spark Toolkits like GraphX or MLlib in the streaming context? Apologies if this is a stupid question but I am