Application kill from UI do not propagate exception

[Noorul Islam K M]

Hi all,

I am trying to trap UI kill event of a spark application from driver.
Some how the exception thrown is not propagated to the driver main
program. See for example using spark-shell below.

Is there a way to get hold of this event and shutdown the driver program?

Regards,
Noorul


spark@spark1:~/spark-2.1.0/sbin$ spark-shell --master
spark://10.29.83.162:7077
Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
17/03/23 15:16:47 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where
applicable
17/03/23 15:16:53 WARN ObjectStore: Failed to get database
global_temp, returning NoSuchObjectException
Spark context Web UI available at http://10.29.83.162:4040
Spark context available as 'sc' (master = spark://10.29.83.162:7077,
app id = app-20170323151648-0002).
Spark session available as 'spark'.
Welcome to
    __
 / __/__  ___ _/ /__
_\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0
  /_/

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java
1.8.0_91)
Type in expressions to have them evaluated.
Type :help for more information.

scala> 17/03/23 15:17:28 ERROR StandaloneSchedulerBackend: Application
has been killed. Reason: Master removed our application: KILLED
17/03/23 15:17:28 ERROR Inbox: Ignoring error
org.apache.spark.SparkException: Exiting due to error from cluster
scheduler: Master removed our application: KILLED
at
org.apache.spark.scheduler.TaskSchedulerImpl.error(TaskSchedulerImpl.scala:459)
at
org.apache.spark.scheduler.cluster.StandaloneSchedulerBackend.dead(StandaloneSchedulerBackend.scala:139)
at
org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint.markDead(StandaloneAppClient.scala:254)
at
org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint$$anonfun$receive$1.applyOrElse(StandaloneAppClient.scala:168)
at
org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:117)
at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)
at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)
at
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:213)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)


scala> sc
res0: org.apache.spark.SparkContext = org.apache.spark.SparkContext@25b8f9d2

scala>



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Application-kill-from-UI-do-not-propagate-exception-tp28539.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

This is a test mail, please ignore!

[Noorul Islam K M]

Sending plain text mail to test whether my mail appear in the list.



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/This-is-a-test-mail-please-ignore-tp28538.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: spark jobserver

[Noorul Islam K M]

A better forum would be

https://groups.google.com/forum/#!forum/spark-jobserver

or

https://gitter.im/spark-jobserver/spark-jobserver

Regards,
Noorul


Madabhattula Rajesh Kumar  writes:

> Hi,
>
> I am getting below an exception when I start the job-server
>
> ./server_start.sh: line 41: kill: (11482) - No such process
>
> Please let me know how to resolve this error
>
> Regards,
> Rajesh

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

[Noorul Islam K M]

> When Initial jobs have not accepted any resources then what all can be
> wrong? Going through stackoverflow and various blogs does not help. Maybe
> need better logging for this? Adding dev
>

Did you take a look at the spark UI to see your resource availability?

Thanks and Regards
Noorul

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Message loss in streaming even with graceful shutdown

[Noorul Islam K M]

Hi all,

I have a streaming application with batch interval 10 seconds.

val sparkConf = new SparkConf().setAppName("RMQWordCount")
  .set("spark.streaming.stopGracefullyOnShutdown", "true")
val ssc = new StreamingContext(sparkConf, Seconds(10))

I also use reduceByKeyAndWindow() API for aggregation at window interval
of 5 minutes.

But when I send a SIGTERM to the streaming process at around 4th minute,
I don't see reduceByKeyAndWindow() action taking place. But the data is
already read for 4 minutes. I thought graceful shutdown would trigger
the action with received messages.

Am I missing something?

Thanks and regards
Noorul

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: installing spark-jobserver on cdh 5.7 and yarn

[Noorul Islam K M]

Reza zade  writes:

> Hi
>
> I have set up a cloudera cluster and work with spark. I want to install
> spark-jobserver on it. What should I do?

Maybe you should send this to spark-jobserver mailing list.

https://github.com/spark-jobserver/spark-jobserver#contact

Thanks and Regards
Noorul

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: spark 2.0 home brew package missing

[Noorul Islam K M]

kalkimann  writes:

> Hi,
> spark 1.6.2 is the latest brew package i can find.
> spark 2.0.x brew package is missing, best i know.
>
> Is there a schedule when spark-2.0 will be available for "brew install"? 
>

Did you do a 'brew update' before searching. I installed spark-2.0 this
week.

Regards
Noorul

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Testing --supervise flag

[Noorul Islam K M]

Hi all,

I was trying to test --supervise flag of spark-submit.

The documentation [1] says that, the flag helps in restarting your
application automatically if it exited with non-zero exit code.

I am looking for some clarification on that documentation. In this
context, does application means the driver?

Will the driver be re-launched if an exception is thrown by the
application? I tested this scenario and the driver is not re-launched.

~/spark-1.6.1/bin/spark-submit --deploy-mode cluster --master 
spark://10.29.83.162:6066 --class 
org.apache.spark.examples.ExceptionHandlingTest 
/home/spark/spark-1.6.1/lib/spark-examples-1.6.1-hadoop2.6.0.jar

I killed the driver java process using 'kill -9' command and the driver
is re-launched. 

Is this the only scenario were driver will be re-launched? Is there a
way to simulate non-zero exit code and test the use of --supervise flag?

Regards,
Noorul

[1] 
http://spark.apache.org/docs/latest/spark-standalone.html#launching-spark-applications

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

When worker is killed driver continues to run causing issues in supervise mode

[Noorul Islam K M]

Spark version: 1.6.1
Cluster Manager: Standalone

I am experimenting with cluster mode deployment along with supervise for
high availability of streaming applications.

1. Submit a streaming job in cluster mode with supervise
2. Say that driver is scheduled on worker1. The app started
   successfully.
3. Kill worker1 java process. This does not kill driver process and
   hence the application (context) is still alive.
4. Because of supervise flag, driver gets scheduled to new worker
   worker2 and hence a new context is created, making it a duplicate.

I think this seems to be a bug.

Regards,
Noorul

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Stage shows incorrect output size

[Noorul Islam K M]

Hi all,

I am trying to copy data from one cassandra cluster to another using
spark + cassandra connector. At the source I have around 200 GB of data
But while running the spark stage shows output as 406 GB and the data is
still getting copied. I wonder why is it showing this high a number.


Environment:

DSE - 4.8.3
Spark - 1.4.1
Spark Cassandra connector - 1.4.1

Thanks and Regards
Noorul

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: error writing to stdout

[Noorul Islam K M]

carlilek  writes:

> My users use Spark 1.5.1 in standalone mode on an HPC cluster, with a
> smattering still using 1.4.0
>
> I have been getting reports of errors like this: 
>
> 15/12/21 15:40:33 ERROR FileAppender: Error writing stream to file
> /scratch/spark/work/app-20151221150645-/3/stdout
> java.io.IOException: Stream closed
>   at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
>   at java.io.BufferedInputStream.read1(BufferedInputStream.java:272)
>   at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>   at java.io.FilterInputStream.read(FilterInputStream.java:107)
>   at
> org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70)
>   at
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39)
>   at
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
>   at
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
>   at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1699)
>   at
> org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38)
> '
>
> So far I have been unable to reproduce reliably, but does anyone have any
> ideas? 
>

I have seen this happening in our cluster also. So far I have been
ignoring this.

Thanks and Regards
Noorul

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Cassandra Connection Issue with Spark-jobserver

[Noorul Islam K M]

Are you using DSE spark, if so are you pointing spark job server to use DSE 
spark?

Thanks and Regards
Noorul

Anand anand.vi...@monotype.com writes:

 *I am new to Spark world and Job Server

 My Code :*

 package spark.jobserver

 import java.nio.ByteBuffer

 import scala.collection.JavaConversions._
 import scala.collection.mutable.ListBuffer
 import scala.collection.immutable.Map

 import org.apache.cassandra.hadoop.ConfigHelper
 import org.apache.cassandra.hadoop.cql3.CqlPagingInputFormat
 import org.apache.cassandra.hadoop.cql3.CqlConfigHelper
 import org.apache.cassandra.hadoop.cql3.CqlOutputFormat
 import org.apache.cassandra.utils.ByteBufferUtil
 import org.apache.hadoop.mapreduce.Job

 import com.typesafe.config.{Config, ConfigFactory}
 import org.apache.spark._
 import org.apache.spark.SparkContext._
 import scala.util.Try

 object CassandraCQLTest extends SparkJob{

   def main(args: Array[String]) {   
 val sc = new SparkContext(local[4], CassandraCQLTest)

 sc.addJar(/extra_data/spark-cassandra-connector/spark-cassandra-connector-java/target/scala-2.10/spark-cassandra-connector-java-assembly-1.3.0-SNAPSHOT.jar);
 val config = ConfigFactory.parseString()
 val results = runJob(sc, config)
 println(Result is  + test)
   }
   
   override def validate(sc: SparkContext, config: Config):
 SparkJobValidation = {
 Try(config.getString(input.string))
   .map(x = SparkJobValid)
   .getOrElse(SparkJobInvalid(No input.string config param))
   }
   
   override def runJob(sc: SparkContext, config: Config): Any = {
 val cHost: String = localhost
 val cPort: String = 9160
 val KeySpace = retail
 val InputColumnFamily = ordercf
 val OutputColumnFamily = salecount

 val job = new Job()
 job.setInputFormatClass(classOf[CqlPagingInputFormat])
 ConfigHelper.setInputInitialAddress(job.getConfiguration(), cHost)
 ConfigHelper.setInputRpcPort(job.getConfiguration(), cPort)
 ConfigHelper.setInputColumnFamily(job.getConfiguration(), KeySpace,
 InputColumnFamily)
 ConfigHelper.setInputPartitioner(job.getConfiguration(),
 Murmur3Partitioner)
 CqlConfigHelper.setInputCQLPageRowSize(job.getConfiguration(), 3)

 /** CqlConfigHelper.setInputWhereClauses(job.getConfiguration(),
 user_id='bob') */

 /** An UPDATE writes one or more columns to a record in a Cassandra
 column family */
 val query = UPDATE  + KeySpace + . + OutputColumnFamily +  SET
 sale_count = ? 
 CqlConfigHelper.setOutputCql(job.getConfiguration(), query)

 job.setOutputFormatClass(classOf[CqlOutputFormat])
 ConfigHelper.setOutputColumnFamily(job.getConfiguration(), KeySpace,
 OutputColumnFamily)
 ConfigHelper.setOutputInitialAddress(job.getConfiguration(), cHost)
 ConfigHelper.setOutputRpcPort(job.getConfiguration(), cPort)
 ConfigHelper.setOutputPartitioner(job.getConfiguration(),
 Murmur3Partitioner)

 val casRdd = sc.newAPIHadoopRDD(job.getConfiguration(),
   classOf[CqlPagingInputFormat],
   classOf[java.util.Map[String,ByteBuffer]],
   classOf[java.util.Map[String,ByteBuffer]])

 
 val productSaleRDD = casRdd.map {
   case (key, value) = {
 (ByteBufferUtil.string(value.get(prod_id)),
 ByteBufferUtil.toInt(value.get(quantity)))
   }
 }
 val aggregatedRDD = productSaleRDD.reduceByKey(_ + _)
 aggregatedRDD.collect().foreach {
   case (productId, saleCount) = println(productId + : + saleCount)
 }

 val casoutputCF  = aggregatedRDD.map {
   case (productId, saleCount) = {
 val outColFamKey = Map(prod_id - ByteBufferUtil.bytes(productId))
 val outKey: java.util.Map[String, ByteBuffer] = outColFamKey
 var outColFamVal = new ListBuffer[ByteBuffer]
 outColFamVal += ByteBufferUtil.bytes(saleCount)
 val outVal: java.util.List[ByteBuffer] = outColFamVal
(outKey, outVal)
   }
 }

 casoutputCF.saveAsNewAPIHadoopFile(
 KeySpace,
 classOf[java.util.Map[String, ByteBuffer]],
 classOf[java.util.List[ByteBuffer]],
 classOf[CqlOutputFormat],
 job.getConfiguration()
   )
 casRdd.count
   }
 }

 *When I push the Jar using spark-jobServer and execute it I get this on
 spark-jobserver terminal
 *
 job-server[ERROR] Exception in thread pool-1-thread-1
 java.lang.NoClassDefFoundError:
 org/apache/cassandra/hadoop/cql3/CqlPagingInputFormat
 job-server[ERROR] at
 spark.jobserver.CassandraCQLTest$.runJob(CassandraCQLTest.scala:46)
 job-server[ERROR] at
 spark.jobserver.CassandraCQLTest$.runJob(CassandraCQLTest.scala:21)
 job-server[ERROR] at
 spark.jobserver.JobManagerActor$$anonfun$spark$jobserver$JobManagerActor$$getJobFuture$4.apply(JobManagerActor.scala:235)
 job-server[ERROR] at
 scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
 job-server[ERROR] at
 scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
 

Re: failed to launch workers on spark

[Noorul Islam K M]

mas mas.ha...@gmail.com writes:

 Hi all!
 I am trying to install spark on my standalone machine. I am able to run the
 master but when i try to run the slaves it gives me following error. Any
 help in this regard will highly be appreciated. 
 _
 localhost: failed to launch org.apache.spark.deploy.worker.Worker:
 localhost: at
 sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
 localhost: at
 sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)


It will be great if you could explain the steps that you followed to
install and run spark.

Also it will be great if you could paste the whole Exception stack.

Thanks and Regards
Noorul

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: What is best way to run spark job in yarn-cluster mode from java program(servlet container) and NOT using spark-submit command.

[Noorul Islam K M]

Sandy Ryza sandy.r...@cloudera.com writes:

 Creating a SparkContext and setting master as yarn-cluster unfortunately
 will not work.

 SPARK-4924 added APIs for doing this in Spark, but won't be included until
 1.4.

 -Sandy


Did you look into something like [1]? With that you can make rest API
call from your java code.

Thanks and Regards
Noorul

[1] https://github.com/spark-jobserver/spark-jobserver?


 On Tue, Mar 17, 2015 at 3:19 AM, Akhil Das ak...@sigmoidanalytics.com
 wrote:

 Create SparkContext set master as yarn-cluster then run it as a standalone
 program?

 Thanks
 Best Regards

 On Tue, Mar 17, 2015 at 1:27 AM, rrussell25 rrussel...@gmail.com wrote:

 Hi, were you ever able to determine a satisfactory approach for this
 problem?
 I have a similar situation and would prefer to execute the job directly
 from
 java code within my jms listener and/or servlet container.



 --
 View this message in context:
 http://apache-spark-user-list.1001560.n3.nabble.com/What-is-best-way-to-run-spark-job-in-yarn-cluster-mode-from-java-program-servlet-container-and-NOT-u-tp21817p22086.html
 Sent from the Apache Spark User List mailing list archive at Nabble.com.

 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org




-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Combining Many RDDs

[Noorul Islam K M]

Yang Chen y...@yang-cs.com writes:

 Hi Noorul,

 Thank you for your suggestion. I tried that, but ran out of memory. I did
 some search and found some suggestions
 that we should try to avoid rdd.union(
 http://stackoverflow.com/questions/28343181/memory-efficient-way-of-union-a-sequence-of-rdds-from-files-in-apache-spark
 ).
 I will try to come up with some other ways.


I think you are using rdd.union(), but I was referring to
SparkContext.union(). I am not sure about the number of RDDs that you
have but I had no issues with memory when I used it to combine 2000
RDDs. Having said that I had other performance issues with spark
cassandra connector.

Thanks and Regards
Noorul


 On Thu, Mar 26, 2015 at 1:13 PM, Noorul Islam K M noo...@noorul.com wrote:

 sparkx y...@yang-cs.com writes:

  Hi,
 
  I have a Spark job and a dataset of 0.5 Million items. Each item performs
  some sort of computation (joining a shared external dataset, if that does
  matter) and produces an RDD containing 20-500 result items. Now I would
 like
  to combine all these RDDs and perform a next job. What I have found out
 is
  that the computation itself is quite fast, but combining these RDDs takes
  much longer time.
 
  val result = data// 0.5M data items
.map(compute(_))   // Produces an RDD - fast
.reduce(_ ++ _)  // Combining RDDs - slow
 
  I have also tried to collect results from compute(_) and use a flatMap,
 but
  that is also slow.
 
  Is there a way to efficiently do this? I'm thinking about writing this
  result to HDFS and reading from disk for the next job, but am not sure if
  that's a preferred way in Spark.
 

 Are you looking for SparkContext.union() [1] ?

 This is not performing well with spark cassandra connector. I am not
 sure whether this will help you.

 Thanks and Regards
 Noorul

 [1]
 http://spark.apache.org/docs/1.3.0/api/scala/index.html#org.apache.spark.SparkContext


-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Combining Many RDDs

[Noorul Islam K M]

sparkx y...@yang-cs.com writes:

 Hi,

 I have a Spark job and a dataset of 0.5 Million items. Each item performs
 some sort of computation (joining a shared external dataset, if that does
 matter) and produces an RDD containing 20-500 result items. Now I would like
 to combine all these RDDs and perform a next job. What I have found out is
 that the computation itself is quite fast, but combining these RDDs takes
 much longer time.

 val result = data// 0.5M data items
   .map(compute(_))   // Produces an RDD - fast
   .reduce(_ ++ _)  // Combining RDDs - slow

 I have also tried to collect results from compute(_) and use a flatMap, but
 that is also slow.

 Is there a way to efficiently do this? I'm thinking about writing this
 result to HDFS and reading from disk for the next job, but am not sure if
 that's a preferred way in Spark.


Are you looking for SparkContext.union() [1] ?

This is not performing well with spark cassandra connector. I am not
sure whether this will help you.

Thanks and Regards
Noorul

[1] 
http://spark.apache.org/docs/1.3.0/api/scala/index.html#org.apache.spark.SparkContext

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: What his the ideal method to interact with Spark Cluster from a Cloud App?

[Noorul Islam K M]

Today I found one answer from a this thread [1] which seems to be worth
exploring.

Michael, if you are reading this, it will be helpful if you could share
more about your spark deployment in production.

Thanks and Regards
Noorul

[1] 
http://apache-spark-user-list.1001560.n3.nabble.com/How-do-you-run-your-spark-app-tp7935p7958.html


Noorul Islam K M noo...@noorul.com writes:

 Hi all,

 We have a cloud application, to which we are adding a reporting service.
 For this we have narrowed down to use Cassandra + Spark for data store
 and processing respectively.

 Since cloud application is separate from Cassandra + Spark deployment,
 what is ideal method to interact with Spark Master from the application?

 We have been evaluating spark-job-server [1], which is an RESTful layer
 on top of Spark.

 Are there any other such tools? Or are there any other better approach
 which can be explored?

 We are evaluating following requirements against spark-job-server,

1. Provide a platform for applications to submit jobs
2. Provide RESTful APIs using which applications will interact with the 
server
   - Upload jar for running jobs
   - Submit job
   - Get job list
   - Get job status
   - Get job result
3. Provide support for kill/restart job
   - Kill job
   - Restart job
4. Support job priority
5. Queue up job submissions if resources not available
6. Troubleshoot job execution
   - Failure – job logs
   - Measure performance
7. Manage cluster deployment
   - Bootstrap, scale up/down (add, remove, replace nodes)
8. Monitor cluster deployment
   - Health report: Report metrics – CPU, Memory, - of jobs, spark 
   processes
   - Alert DevOps about threshold limit of these metrics
   - Alert DevOps about job failures
   - Self healing?
9. Security
   - AAA job submissions
10. High availability/Redundancy
   - This is for the spark-jobserver component itself

 Any help is appreciated!

 Thanks and Regards
 Noorul

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

What his the ideal method to interact with Spark Cluster from a Cloud App?

[Noorul Islam K M]

Hi all,

We have a cloud application, to which we are adding a reporting service.
For this we have narrowed down to use Cassandra + Spark for data store
and processing respectively.

Since cloud application is separate from Cassandra + Spark deployment,
what is ideal method to interact with Spark Master from the application?

We have been evaluating spark-job-server [1], which is an RESTful layer
on top of Spark.

Are there any other such tools? Or are there any other better approach
which can be explored?

We are evaluating following requirements against spark-job-server,

   1. Provide a platform for applications to submit jobs
   2. Provide RESTful APIs using which applications will interact with the 
   server
  - Upload jar for running jobs
  - Submit job
  - Get job list
  - Get job status
  - Get job result
   3. Provide support for kill/restart job
  - Kill job
  - Restart job
   4. Support job priority
   5. Queue up job submissions if resources not available
   6. Troubleshoot job execution
  - Failure – job logs
  - Measure performance
   7. Manage cluster deployment
  - Bootstrap, scale up/down (add, remove, replace nodes)
   8. Monitor cluster deployment
  - Health report: Report metrics – CPU, Memory, - of jobs, spark 
  processes
  - Alert DevOps about threshold limit of these metrics
  - Alert DevOps about job failures
  - Self healing?
   9. Security
  - AAA job submissions
   10. High availability/Redundancy
  - This is for the spark-jobserver component itself

Any help is appreciated!

Thanks and Regards
Noorul

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org