Converting spark batch to spark streaming

I'm trying to convert a spark batch application to a streaming application and wondering what function (or design pattern) I should use to execute a series of operations inside the driver upon arrival of each message (a text file inside an HDFS folder) before starting computation inside executors.

Passing authentication token to the user session in Spark Thrift Server

What's the recommended way of associating authentication token (response to a successful login) to the user session from a custom authenticator (PasswdAuthenticationProvider)? Thanks, Mohammad

SparkLauncher reliability and scalability

We are using SparkLauncher and SparkAppHandle.Listener to launch spark applications from a Java web application and listen to the state changes. Our observation is that as the number of concurrent jobs grow sometimes some of the state changes are not reported (e.g. some applications never report fi