Hi Lakshman, We’ve set these 2 properties to bringup spark history server
spark.history.fs.logDirectory <location where the logs are stored> spark.history.ui.port <Port on which history server should come up> We’re writing the logs to HDFS. In order to write logs, we’re setting following properties while submitting the spark job spark.eventLog.enabled true spark.eventLog.dir <Directory where logs should be written> Thanks and Regards, Abhishek From: Battini Lakshman <battini.laksh...@gmail.com> Sent: Wednesday, January 23, 2019 1:55 PM To: Rao, Abhishek (Nokia - IN/Bangalore) <abhishek....@nokia.com> Subject: Re: Spark UI History server on Kubernetes HI Abhishek, Thank you for your response. Could you please let me know the properties you configured for bringing up History Server and its UI. Also, are you writing the logs to any directory on persistent storage, if yes, could you let me know the changes you did in Spark to write logs to that directory. Thanks! Best Regards, Lakshman Battini. On Tue, Jan 22, 2019 at 10:53 PM Rao, Abhishek (Nokia - IN/Bangalore) <abhishek....@nokia.com<mailto:abhishek....@nokia.com>> wrote: Hi, We’ve setup spark-history service (based on spark 2.4) on K8S. UI works perfectly fine when running on NodePort. We’re facing some issues when on ingress. Please let us know what kind of inputs do you need? Thanks and Regards, Abhishek From: Battini Lakshman <battini.laksh...@gmail.com<mailto:battini.laksh...@gmail.com>> Sent: Tuesday, January 22, 2019 6:02 PM To: user@spark.apache.org<mailto:user@spark.apache.org> Subject: Spark UI History server on Kubernetes Hello, We are running Spark 2.4 on Kubernetes cluster, able to access the Spark UI using "kubectl port-forward". However, this spark UI contains currently running Spark application logs, we would like to maintain the 'completed' spark application logs as well. Could someone help us to setup 'Spark History server' on Kubernetes. Thanks! Best Regards, Lakshman Battini.
