Getting FileNotFoundException and LeaseExpired Exception while writing a df to hdfs path

Hi I am receiving FileNotFoundException and LeaseExpired Exception while writing a data frame to an hdfs path.I am using spark 1.6 and reading messages from Tibco in my streaming application .I am doing some tranformations on each Rdd and converting it to a data frame and writing to an hdfs path.

Packaging kafka certificates in uber jar

I've been trying to read from kafka via a spark streaming client. I found out spark cluster doesn't have certificates deployed. Then I tried using the same local certificates I've been testing with by packing them in an uber jar and getting a File handle from the Classloader resource. But I'm getti

Re: Questions about caching

Hi Andrew, Since you mentioned the alternative solution with Alluxio , here is a more comprehensive tutorial on caching Spark dataframes on Alluxio: https://www.alluxio.com/blog/effective-spark-dataframes-with-alluxio Namely, caching your dataframe is simply running df.write.p