Re: Kill Spark Streaming JOB from Spark UI or Yarn

[Matei Zaharia]

The batches should all have the same application ID, so use that one. You can 
also find the application in the YARN UI to terminate it from there.

Matei

> On Aug 27, 2017, at 10:27 AM, KhajaAsmath Mohammed  
> wrote:
> 
> Hi,
> 
> I am new to spark streaming and not able to find an option to kill it after 
> starting spark streaming context.
> 
> Streaming Tab doesnt have option to kill it.
> 
> Jobs tab too doesn't have option to kill it
> 
> 
> 
> if scheduled on yarn, how to kill that if spark submit is running in 
> background as I will not have an option to find yarn application id. does 
> batches have separate yarn application id or same one?
> 
> Thanks,
> Asmath


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: different behaviour linux/Unix vs windows when load spark context in scala method called from R function using rscala package

[Georg Heiler]

Why don't you simply use sparklyr for a more R native integration of spark?
Simone Pallotta  schrieb am So. 27. Aug.
2017 um 09:47:

> In my R code, I am using rscala package to bridge to a scala method. in
> scala method I have initialized a spark context to be used later.
>
>
> R code:
>
> s <- scala(classpath = "", heap.maximum = "4g")
>
> assign("WrappeR",s$.it.wrapper.r.Wrapper)
>
> WrappeR$init()
>
> where init is a scala function and Wrapper is scala class name (see below).
>
> I created a scala object (Wrapper) with init function
>
>
> scala code:
>
> package it.wrapper.r
>
> object Wrapper {
>
>   var Spark_context: SparkContext = _
>
> def init(): Unit = {
>
> val spark_conf = new SparkConf()
>
>   .setMaster("local[*]")
>
>   .setAppName("GMQL-R")
>
>   .set("spark.serializer",
> "org.apache.spark.serializer.KryoSerializer")
>
>   .set("spark.executor.memory", "6g")
>
>   .set("spark.driver.memory", "2g")
>
> Spark_context = SparkContext.getOrCreate(spark_conf)
>
>   }
>
> }
>
> I removed the following code after Spark_context because is not useful.
>
> this code works without flaws under OS X and Linux. Of course window is
> not gentle as other OSes. I checked out my project under windows (I have
> installed scala 2.11.8, java 1.8 and create environment variable JAVA_HOME
> and SCALA_HOME that are mandatory for running rscala package in R, I did
> the same in OS X/Linux)
>
> when I run the jar file as standalone application it works fine under
> windows, but when I invoke the same function from R it fails giving this
> error:
>
> Caused by: java.lang.RuntimeException: java.lang.RuntimeException: class
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not
> org.apache.hadoop.security.GroupMappingServiceProvider
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1752)
>
> at org.apache.hadoop.security.Groups.(Groups.java:55)
>
> at
> org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:182)
>
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:235)
>
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:214)
>
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:669)
>
> at `enter code
> here`org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571)
>
> at
> org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2391)
>
> at
> org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2391)
>
> at scala.Option.getOrElse(Option.scala:121)
>
> at
> org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2391)
>
> at org.apache.spark.SparkContext.(SparkContext.scala:295)
>
> at org.apache.spark.SparkContext.(SparkContext.scala:145)
>
> at it.polimi.genomics.r.Wrapper$.initGMQL(Wrapper.scala:98)
>
> at $line4.$read$$iw$$iw$$anonfun$1.apply$mcV$sp(:19)
>
> at $line4.$read$$iw$$iw$$anonfun$1.apply(:16)
>
> at $line4.$read$$iw$$iw$$anonfun$1.apply(:16)
>
> ... 46 more
>
> Caused by: java.lang.RuntimeException: class
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not
> org.apache.hadoop.security.GroupMappingServiceProvider
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1746)
>
> ... 62 more
>
> java.lang.reflect.InvocationTargetException
>
> java.lang.RuntimeException: java.lang.RuntimeException: class
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not
> org.apache.hadoop.security.GroupMappingServiceProvider
>
>
> Does anyone has already seen this kind of behavior?
>

different behaviour linux/Unix vs windows when load spark context in scala method called from R function using rscala package

[Simone Pallotta]

In my R code, I am using rscala package to bridge to a scala method. in scala 
method I have initialized a spark context to be used later.


R code:

s <- scala(classpath = "", heap.maximum = "4g")

assign("WrappeR",s$.it.wrapper.r.Wrapper)

WrappeR$init()


where init is a scala function and Wrapper is scala class name (see below).

I created a scala object (Wrapper) with init function


scala code:

package it.wrapper.r

object Wrapper {

  var Spark_context: SparkContext = _

def init(): Unit = {

val spark_conf = new SparkConf()

  .setMaster("local[*]")

  .setAppName("GMQL-R")

  .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")

  .set("spark.executor.memory", "6g")

  .set("spark.driver.memory", "2g")

Spark_context = SparkContext.getOrCreate(spark_conf)

  }

}

I removed the following code after Spark_context because is not useful.

this code works without flaws under OS X and Linux. Of course window is not 
gentle as other OSes. I checked out my project under windows (I have installed 
scala 2.11.8, java 1.8 and create environment variable JAVA_HOME and SCALA_HOME 
that are mandatory for running rscala package in R, I did the same in OS 
X/Linux)

when I run the jar file as standalone application it works fine under windows, 
but when I invoke the same function from R it fails giving this error:


Caused by: java.lang.RuntimeException: java.lang.RuntimeException: class 
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not 
org.apache.hadoop.security.GroupMappingServiceProvider

at 
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1752)

at org.apache.hadoop.security.Groups.(Groups.java:55)

at 
org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:182)

at 
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:235)

at 
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:214)

at 
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:669)

at `enter code 
here`org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571)

at 
org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2391)

at 
org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2391)

at scala.Option.getOrElse(Option.scala:121)

at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2391)

at org.apache.spark.SparkContext.(SparkContext.scala:295)

at org.apache.spark.SparkContext.(SparkContext.scala:145)

at it.polimi.genomics.r.Wrapper$.initGMQL(Wrapper.scala:98)

at $line4.$read$$iw$$iw$$anonfun$1.apply$mcV$sp(:19)

at $line4.$read$$iw$$iw$$anonfun$1.apply(:16)

at $line4.$read$$iw$$iw$$anonfun$1.apply(:16)

... 46 more

Caused by: java.lang.RuntimeException: class 
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not 
org.apache.hadoop.security.GroupMappingServiceProvider

at 
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1746)

... 62 more

java.lang.reflect.InvocationTargetException

java.lang.RuntimeException: java.lang.RuntimeException: class 
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not 
org.apache.hadoop.security.GroupMappingServiceProvider


Does anyone has already seen this kind of behavior?