We have few Spark Streaming Apps running on our AWS Spark 2.1 Yarn cluster.
We currently log on to the Master Node of the cluster and start the App
using "spark-submit", calling the jar.
We would like to open up this to our users, so that they can submit their
own Apps, but we would not be able to give Users access to the Master or
any other Nodes of the cluster. I have the below questions.
- What would be a good interface to invoke the spark-submit in this
case? I read about Livy & jobserver but couldn't find out if it provides a
UI.
- Is there a way to specify the start-date and end-date for the
streaming Apps? Option to know the current App status and option to Kill
App. I know Resource Manager provides this but not sure if it is a good
idea to open up RM to users.
- Is there a way to control Access to the App such that User-A can only
execute App-1 and App-3?
- Is there a way to control Resources usage such that User-A can only
use 2 Executors, 2 Cores/Executor, 10GB/Executor etc:?
Thank you.
Ajit
